2020-03-11T22:22:48.357 INFO:root:teuthology version: 1.0.0-1def34b 2020-03-11T22:22:48.358 DEBUG:teuthology.run:Teuthology command: teuthology /tmp/teuthology-worker.2Df5zZ.tmp -- --verbose --lock --description rados/standalone/{supported-random-distro$/{centos_8.yaml} workloads/osd.yaml} --name dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi --block --owner scheduled_dzafman@bionic-laptop --archive /home/teuthworker/archive/dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi/4848259 2020-03-11T22:22:48.713 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-11T22:22:48.792 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi/4848259 branch: wip-zafman-octo-diag description: rados/standalone/{supported-random-distro$/{centos_8.yaml} workloads/osd.yaml} email: dzafman@redhat.com first_in_suite: false job_id: '4848259' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: centos os_version: '8.1' overrides: admin_socket: branch: wip-zafman-octo-diag ceph: conf: mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 25 log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-zafman-octo-diag sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 owner: scheduled_dzafman@bionic-laptop priority: 20 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 suite: rados suite_branch: wip-zafman-octo-diag suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-zafman-octo-diag/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 tasks: - install: null - workunit: basedir: qa/standalone clients: all: - osd teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.483 2020-03-11T22:22:48.792 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-zafman-octo-diag/qa; will attempt to use it 2020-03-11T22:22:48.796 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-zafman-octo-diag/qa/tasks 2020-03-11T22:22:48.797 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-03-11T22:22:48.799 INFO:teuthology.task.internal:Checking packages... 2020-03-11T22:22:48.827 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash '4acc1121e47039a3cdf2978ae458c10d153aa141' 2020-03-11T22:22:48.827 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-03-11T22:22:48.828 INFO:teuthology.packaging:ref: None 2020-03-11T22:22:48.828 INFO:teuthology.packaging:tag: None 2020-03-11T22:22:48.828 INFO:teuthology.packaging:branch: wip-zafman-octo-diag 2020-03-11T22:22:48.828 INFO:teuthology.packaging:sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-11T22:22:48.828 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-zafman-octo-diag 2020-03-11T22:22:49.105 INFO:teuthology.task.internal:Found packages for ceph version 15.1.0-1941.g4acc112 2020-03-11T22:22:49.105 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-03-11T22:22:49.110 INFO:teuthology.task.internal:no buildpackages task found 2020-03-11T22:22:49.111 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-03-11T22:22:49.195 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-03-11T22:22:49.196 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-11T22:22:49.240 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2020-03-11T22:22:59.260 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2020-03-11T22:23:09.279 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2020-03-11T22:23:19.297 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2020-03-11T22:23:29.336 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2020-03-11T22:23:39.351 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2020-03-11T22:23:49.367 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2020-03-11T22:23:59.383 DEBUG:teuthology.lock.ops:lock_many request: {'count': 1, 'locked_by': 'scheduled_dzafman@bionic-laptop', 'description': '/home/teuthworker/archive/dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi/4848259', 'machine_type': 'smithi'} 2020-03-11T22:23:59.400 DEBUG:teuthology.lock.ops:locked ubuntu@smithi145.front.sepia.ceph.com 2020-03-11T22:23:59.430 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2020-03-11T22:23:59.620 INFO:teuthology.provision.fog.smithi145:Scheduling deploy of centos 8.1 2020-03-11T22:23:59.945 INFO:teuthology.orchestra.console:Power off smithi145 2020-03-11T22:23:59.945 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-03-11T22:23:59.998 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:24:04.026 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:24:08.138 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:24:12.251 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:24:12.461 INFO:teuthology.orchestra.console:Power off for smithi145 completed 2020-03-11T22:24:12.562 INFO:teuthology.orchestra.console:Power on smithi145 2020-03-11T22:24:12.562 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-03-11T22:24:12.582 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:24:16.607 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:24:16.819 INFO:teuthology.orchestra.console:Power on for smithi145 completed 2020-03-11T22:24:16.920 INFO:teuthology.provision.fog.smithi145:Waiting for deploy to finish 2020-03-11T22:26:47.812 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi145.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:26:47.872 ERROR:teuthology.orchestra.connection:Error connecting to smithi145.front.sepia.ceph.com Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/connection.py", line 106, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/client.py", line 446, in connect passphrase, File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/client.py", line 764, in _auth raise saved_exception AuthenticationException: Authentication failed. 2020-03-11T22:27:54.878 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi145.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:29:00.880 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi145.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:29:01.145 INFO:teuthology.orchestra.run.smithi145:> hostname 2020-03-11T22:29:01.422 INFO:teuthology.orchestra.run.smithi145.stdout:smithi145.front.sepia.ceph.com 2020-03-11T22:29:01.423 INFO:teuthology.orchestra.run.smithi145:> grep smithi145.front.sepia.ceph.com /etc/hosts 2020-03-11T22:29:01.477 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.15.145 smithi145.front.sepia.ceph.com smithi145 2020-03-11T22:29:01.478 INFO:teuthology.orchestra.run.smithi145:> sudo hostname smithi145 2020-03-11T22:29:01.594 INFO:teuthology.orchestra.run.smithi145:> sudo sed -i -e 's/smithi145.front.sepia.ceph.com/smithi145/g' /etc/hosts 2020-03-11T22:29:01.622 INFO:teuthology.orchestra.run.smithi145:> sudo sed -i -e 's/172.21.15.145/172.21.15.145/g' /etc/hosts 2020-03-11T22:29:01.690 INFO:teuthology.orchestra.run.smithi145:> sudo sed -i -e 's/smithi145.front.sepia.ceph.com/smithi145/g' /etc/hostname 2020-03-11T22:29:01.757 INFO:teuthology.orchestra.run.smithi145:> sudo hostname smithi145 2020-03-11T22:29:01.823 INFO:teuthology.provision.fog.smithi145:Deploy complete! 2020-03-11T22:29:01.825 DEBUG:teuthology.parallel:result is None 2020-03-11T22:29:01.972 INFO:teuthology.lock.ops:Checking smithi145.front.sepia.ceph.com 2020-03-11T22:29:01.972 INFO:teuthology.lock.ops:New key found. Updating... 2020-03-11T22:29:01.990 INFO:teuthology.lock.ops:Updating [smithi145.front.sepia.ceph.com]: set os type and version on server 2020-03-11T22:29:01.991 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:29:01.991 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi145.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:29:02.061 INFO:teuthology.orchestra.run.smithi145:> true 2020-03-11T22:29:02.220 INFO:teuthology.orchestra.run.smithi145:> uname -m 2020-03-11T22:29:02.273 INFO:teuthology.orchestra.run.smithi145.stdout:x86_64 2020-03-11T22:29:02.273 INFO:teuthology.orchestra.run.smithi145:> cat /etc/os-release 2020-03-11T22:29:02.326 INFO:teuthology.orchestra.run.smithi145.stdout:NAME="CentOS Linux" 2020-03-11T22:29:02.326 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION="8 (Core)" 2020-03-11T22:29:02.327 INFO:teuthology.orchestra.run.smithi145.stdout:ID="centos" 2020-03-11T22:29:02.327 INFO:teuthology.orchestra.run.smithi145.stdout:ID_LIKE="rhel fedora" 2020-03-11T22:29:02.327 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION_ID="8" 2020-03-11T22:29:02.327 INFO:teuthology.orchestra.run.smithi145.stdout:PLATFORM_ID="platform:el8" 2020-03-11T22:29:02.327 INFO:teuthology.orchestra.run.smithi145.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-03-11T22:29:02.328 INFO:teuthology.orchestra.run.smithi145.stdout:ANSI_COLOR="0;31" 2020-03-11T22:29:02.328 INFO:teuthology.orchestra.run.smithi145.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-03-11T22:29:02.328 INFO:teuthology.orchestra.run.smithi145.stdout:HOME_URL="https://www.centos.org/" 2020-03-11T22:29:02.328 INFO:teuthology.orchestra.run.smithi145.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-03-11T22:29:02.328 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:29:02.328 INFO:teuthology.orchestra.run.smithi145.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-03-11T22:29:02.328 INFO:teuthology.orchestra.run.smithi145.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-03-11T22:29:02.329 INFO:teuthology.orchestra.run.smithi145.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-03-11T22:29:02.329 INFO:teuthology.orchestra.run.smithi145.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-03-11T22:29:02.329 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:29:02.330 INFO:teuthology.lock.ops:Updating smithi145.front.sepia.ceph.com on lock server 2020-03-11T22:29:02.344 INFO:teuthology.task.internal.lock_machines:1 smithi machines locked this try, 1/1 locked so far 2020-03-11T22:29:02.360 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi145.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDfZe6JAgc3iRM5sqQd8fNcBPdUXwIh3KSQX45U152yvxp30itVgUUYpfcnvKpDnXdP7dq1Ni1k63cWsunaSvgpIsuSyFMPN8+PAOHiTzQtdApZqD1YN3gsJjMDJDqj1lhbYvrZ8EoXJhrjSiWw6pSkaK3BdssjWIW81L4wn8D3N6nO9DJxBPFPvE7iPyc85jvwgiY3SrE/VZa10jn8N2zl4FhA4z0rsy4sYGHMXlAWj+rc/DN+4M/H7jZiyh8ffAsiM9Y14qtn40MkpJBC1Eqin4Wu1gPIk/oZ22TK3karE4GN6YXaSB7ZRaWu3rRIX6TOfBRYlRebsNiwbkZdxY5JQXYjY1d46WqLh77C5W9olvfkByp7vkEJKfd6X93GzXCl/PpPaJo1MFBae1BweBq8nEYLvVh1/y9RPf2QuiizQTH4eMgMq94MJZdeA54O2VqlsFZePQDJFW8ObRuaKtmTw+yzaTW5msgcsrELhEr/8Ed+ym4czcntnEOv7AFaMis= 2020-03-11T22:29:02.360 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-11T22:29:02.402 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-03-11T22:29:02.408 INFO:teuthology.task.internal:Saving configuration 2020-03-11T22:29:02.442 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-03-11T22:29:02.450 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-03-11T22:29:02.465 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2020-03-11 22:23:59.390150', u'locked_by': u'scheduled_dzafman@bionic-laptop', u'up': True, u'mac_address': None, u'name': u'smithi145.front.sepia.ceph.com', u'os_version': u'8', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'centos', u'arch': u'x86_64', u'ssh_pub_key': u'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOBdcXcP5VUOmzyL5aVJChGv99dNajqd18vs6NF6k8y1VJqEE+D+TCQt6chMv37gOE/k5foK9bmplfFx945gbu0=', u'description': u'/home/teuthworker/archive/dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi/4848259'} 2020-03-11T22:29:02.465 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-03-11T22:29:02.545 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2020-03-11T22:29:02.546 INFO:teuthology.run_tasks:Running task console_log... 2020-03-11T22:29:02.591 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2020-03-11T22:29:02.591 INFO:teuthology.run_tasks:Running task internal.connect... 2020-03-11T22:29:02.603 INFO:teuthology.task.internal:Opening connections... 2020-03-11T22:29:02.603 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi145.front.sepia.ceph.com 2020-03-11T22:29:02.605 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi145.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:29:02.677 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-03-11T22:29:02.687 INFO:teuthology.orchestra.run.smithi145:> uname -m 2020-03-11T22:29:02.836 INFO:teuthology.orchestra.run.smithi145.stdout:x86_64 2020-03-11T22:29:02.837 INFO:teuthology.orchestra.run.smithi145:> cat /etc/os-release 2020-03-11T22:29:02.890 INFO:teuthology.orchestra.run.smithi145.stdout:NAME="CentOS Linux" 2020-03-11T22:29:02.890 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION="8 (Core)" 2020-03-11T22:29:02.890 INFO:teuthology.orchestra.run.smithi145.stdout:ID="centos" 2020-03-11T22:29:02.890 INFO:teuthology.orchestra.run.smithi145.stdout:ID_LIKE="rhel fedora" 2020-03-11T22:29:02.890 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION_ID="8" 2020-03-11T22:29:02.890 INFO:teuthology.orchestra.run.smithi145.stdout:PLATFORM_ID="platform:el8" 2020-03-11T22:29:02.891 INFO:teuthology.orchestra.run.smithi145.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-03-11T22:29:02.891 INFO:teuthology.orchestra.run.smithi145.stdout:ANSI_COLOR="0;31" 2020-03-11T22:29:02.891 INFO:teuthology.orchestra.run.smithi145.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-03-11T22:29:02.891 INFO:teuthology.orchestra.run.smithi145.stdout:HOME_URL="https://www.centos.org/" 2020-03-11T22:29:02.891 INFO:teuthology.orchestra.run.smithi145.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-03-11T22:29:02.891 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:29:02.892 INFO:teuthology.orchestra.run.smithi145.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-03-11T22:29:02.892 INFO:teuthology.orchestra.run.smithi145.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-03-11T22:29:02.892 INFO:teuthology.orchestra.run.smithi145.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-03-11T22:29:02.892 INFO:teuthology.orchestra.run.smithi145.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-03-11T22:29:02.892 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:29:02.892 INFO:teuthology.lock.ops:Updating smithi145.front.sepia.ceph.com on lock server 2020-03-11T22:29:02.905 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-03-11T22:29:02.942 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-03-11T22:29:02.952 INFO:teuthology.task.internal:Checking for old test directory... 2020-03-11T22:29:02.952 INFO:teuthology.orchestra.run.smithi145:> test '!' -e /home/ubuntu/cephtest 2020-03-11T22:29:02.965 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-03-11T22:29:02.975 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-03-11T22:29:02.976 INFO:teuthology.orchestra.run.smithi145:> test -z $(ls -A /var/lib/ceph) 2020-03-11T22:29:03.038 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-03-11T22:29:03.071 INFO:teuthology.run_tasks:Running task kernel... 2020-03-11T22:29:03.095 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'osd.2': {'sha1': 'distro', 'kdb': True}, 'osd.1': {'sha1': 'distro', 'kdb': True}, 'osd.0': {'sha1': 'distro', 'kdb': True}, 'client.0': {'sha1': 'distro', 'kdb': True}, 'mgr.x': {'sha1': 'distro', 'kdb': True}} 2020-03-11T22:29:03.095 INFO:teuthology.task.kernel:config {'mon.a': {'sha1': 'distro', 'kdb': True}}, timeout 300 2020-03-11T22:29:03.095 INFO:teuthology.orchestra.run.smithi145:> uname -r 2020-03-11T22:29:03.110 INFO:teuthology.orchestra.run.smithi145.stdout:4.18.0-147.5.1.el8_1.x86_64 2020-03-11T22:29:03.110 INFO:teuthology.task.kernel:Running kernel on smithi145: 4.18.0-147.5.1.el8_1.x86_64 2020-03-11T22:29:03.110 INFO:teuthology.orchestra.run.smithi145:> sudo yum install -y kernel 2020-03-11T22:29:05.681 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 1 day, 7:03:12 ago on Tue 10 Mar 2020 03:25:53 PM UTC. 2020-03-11T22:29:06.075 INFO:teuthology.orchestra.run.smithi145.stdout:Package kernel-4.18.0-147.el8.x86_64 is already installed. 2020-03-11T22:29:06.075 INFO:teuthology.orchestra.run.smithi145.stdout:Package kernel-4.18.0-147.5.1.el8_1.x86_64 is already installed. 2020-03-11T22:29:06.234 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2020-03-11T22:29:06.235 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2020-03-11T22:29:06.235 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2020-03-11T22:29:06.367 INFO:teuthology.orchestra.run.smithi145:> echo no | sudo yum reinstall kernel || true 2020-03-11T22:29:06.754 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 1 day, 7:03:13 ago on Tue 10 Mar 2020 03:25:53 PM UTC. 2020-03-11T22:29:07.153 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2020-03-11T22:29:07.154 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:29:07.154 INFO:teuthology.orchestra.run.smithi145.stdout: Package Architecture Version Repository Size 2020-03-11T22:29:07.154 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:29:07.154 INFO:teuthology.orchestra.run.smithi145.stdout:Reinstalling: 2020-03-11T22:29:07.154 INFO:teuthology.orchestra.run.smithi145.stdout: kernel x86_64 4.18.0-147.5.1.el8_1 CentOS-Base 1.5 M 2020-03-11T22:29:07.154 INFO:teuthology.orchestra.run.smithi145.stdout: kernel x86_64 4.18.0-147.el8 CentOS-Base 1.5 M 2020-03-11T22:29:07.154 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:29:07.155 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2020-03-11T22:29:07.155 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:29:07.155 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:29:07.156 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 3.1 M 2020-03-11T22:29:07.156 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 0 2020-03-11T22:29:07.156 INFO:teuthology.orchestra.run.smithi145.stderr:Operation aborted. 2020-03-11T22:29:07.244 INFO:teuthology.orchestra.run.smithi145.stdout:Is this ok [y/N]: 2020-03-11T22:29:07.244 INFO:teuthology.orchestra.run.smithi145:> sudo yum reinstall -y kernel || true 2020-03-11T22:29:07.631 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 1 day, 7:03:14 ago on Tue 10 Mar 2020 03:25:53 PM UTC. 2020-03-11T22:29:08.005 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2020-03-11T22:29:08.005 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:29:08.006 INFO:teuthology.orchestra.run.smithi145.stdout: Package Architecture Version Repository Size 2020-03-11T22:29:08.006 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:29:08.006 INFO:teuthology.orchestra.run.smithi145.stdout:Reinstalling: 2020-03-11T22:29:08.006 INFO:teuthology.orchestra.run.smithi145.stdout: kernel x86_64 4.18.0-147.5.1.el8_1 CentOS-Base 1.5 M 2020-03-11T22:29:08.006 INFO:teuthology.orchestra.run.smithi145.stdout: kernel x86_64 4.18.0-147.el8 CentOS-Base 1.5 M 2020-03-11T22:29:08.006 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:29:08.007 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2020-03-11T22:29:08.007 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:29:08.007 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:29:08.007 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 3.1 M 2020-03-11T22:29:08.008 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 0 2020-03-11T22:29:08.008 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2020-03-11T22:29:08.076 INFO:teuthology.orchestra.run.smithi145.stdout:(1/2): kernel-4.18.0-147.5.1.el8_1.x86_64.rpm 24 MB/s | 1.5 MB 00:00 2020-03-11T22:29:08.087 INFO:teuthology.orchestra.run.smithi145.stdout:(2/2): kernel-4.18.0-147.el8.x86_64.rpm 20 MB/s | 1.5 MB 00:00 2020-03-11T22:29:08.088 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:29:08.088 INFO:teuthology.orchestra.run.smithi145.stdout:Total 37 MB/s | 3.1 MB 00:00 2020-03-11T22:29:08.111 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2020-03-11T22:29:08.287 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2020-03-11T22:29:08.287 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2020-03-11T22:29:08.349 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2020-03-11T22:29:08.350 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2020-03-11T22:29:08.495 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2020-03-11T22:29:08.638 INFO:teuthology.orchestra.run.smithi145.stdout: Reinstalling : kernel-4.18.0-147.el8.x86_64 1/4 2020-03-11T22:29:08.764 INFO:teuthology.orchestra.run.smithi145.stdout: Reinstalling : kernel-4.18.0-147.5.1.el8_1.x86_64 2/4 2020-03-11T22:29:08.765 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : kernel-4.18.0-147.el8.x86_64 3/4 2020-03-11T22:29:08.898 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-4.18.0-147.el8.x86_64 3/4 2020-03-11T22:29:08.898 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : kernel-4.18.0-147.5.1.el8_1.x86_64 4/4 2020-03-11T22:29:09.124 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-4.18.0-147.5.1.el8_1.x86_64 4/4 2020-03-11T22:29:09.124 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : kernel-4.18.0-147.5.1.el8_1.x86_64 1/4 2020-03-11T22:29:09.124 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : kernel-4.18.0-147.5.1.el8_1.x86_64 2/4 2020-03-11T22:29:09.124 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : kernel-4.18.0-147.el8.x86_64 3/4 2020-03-11T22:29:09.336 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : kernel-4.18.0-147.el8.x86_64 4/4 2020-03-11T22:29:09.336 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:29:09.336 INFO:teuthology.orchestra.run.smithi145.stdout:Reinstalled: 2020-03-11T22:29:09.336 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-4.18.0-147.5.1.el8_1.x86_64 kernel-4.18.0-147.el8.x86_64 2020-03-11T22:29:09.336 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:29:09.337 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2020-03-11T22:29:09.436 INFO:teuthology.orchestra.run.smithi145:> rpm -q kernel --last | head -n 1 2020-03-11T22:29:09.490 INFO:teuthology.orchestra.run.smithi145.stdout:kernel-4.18.0-147.el8.x86_64 Wed 11 Mar 2020 10:29:08 PM UTC 2020-03-11T22:29:09.491 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-147.el8.x86_64 2020-03-11T22:29:09.491 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-147.5.1.el8_1.x86_64 Expected: 4.18.0-147.el8.x86_64 2020-03-11T22:29:09.491 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2020-03-11T22:29:09.492 DEBUG:teuthology.task.kernel:src is distro, skipping download 2020-03-11T22:29:09.492 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2020-03-11T22:29:09.492 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi145.front.sepia.ceph.com, path=None, version=distro) 2020-03-11T22:29:09.492 INFO:teuthology.orchestra.run.smithi145:> rpm -q kernel --last | head -n 1 2020-03-11T22:29:09.533 INFO:teuthology.orchestra.run.smithi145.stdout:kernel-4.18.0-147.el8.x86_64 Wed 11 Mar 2020 10:29:08 PM UTC 2020-03-11T22:29:09.535 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-147.el8.x86_64 2020-03-11T22:29:09.535 INFO:teuthology.orchestra.run.smithi145:> sudo rpm -qi grub2-tools 2020-03-11T22:29:09.606 INFO:teuthology.orchestra.run.smithi145.stdout:Name : grub2-tools 2020-03-11T22:29:09.606 INFO:teuthology.orchestra.run.smithi145.stdout:Epoch : 1 2020-03-11T22:29:09.606 INFO:teuthology.orchestra.run.smithi145.stdout:Version : 2.02 2020-03-11T22:29:09.606 INFO:teuthology.orchestra.run.smithi145.stdout:Release : 78.el8 2020-03-11T22:29:09.607 INFO:teuthology.orchestra.run.smithi145.stdout:Architecture: x86_64 2020-03-11T22:29:09.607 INFO:teuthology.orchestra.run.smithi145.stdout:Install Date: Tue 10 Mar 2020 03:09:09 PM UTC 2020-03-11T22:29:09.607 INFO:teuthology.orchestra.run.smithi145.stdout:Group : System Environment/Base 2020-03-11T22:29:09.607 INFO:teuthology.orchestra.run.smithi145.stdout:Size : 9192691 2020-03-11T22:29:09.607 INFO:teuthology.orchestra.run.smithi145.stdout:License : GPLv3+ 2020-03-11T22:29:09.607 INFO:teuthology.orchestra.run.smithi145.stdout:Signature : RSA/SHA256, Wed 04 Dec 2019 11:50:58 PM UTC, Key ID 05b555b38483c65d 2020-03-11T22:29:09.607 INFO:teuthology.orchestra.run.smithi145.stdout:Source RPM : grub2-2.02-78.el8.src.rpm 2020-03-11T22:29:09.608 INFO:teuthology.orchestra.run.smithi145.stdout:Build Date : Mon 25 Nov 2019 04:08:30 PM UTC 2020-03-11T22:29:09.608 INFO:teuthology.orchestra.run.smithi145.stdout:Build Host : kbuilder.bsys.centos.org 2020-03-11T22:29:09.608 INFO:teuthology.orchestra.run.smithi145.stdout:Relocations : (not relocatable) 2020-03-11T22:29:09.608 INFO:teuthology.orchestra.run.smithi145.stdout:Packager : CentOS BuildSystem 2020-03-11T22:29:09.608 INFO:teuthology.orchestra.run.smithi145.stdout:Vendor : CentOS 2020-03-11T22:29:09.608 INFO:teuthology.orchestra.run.smithi145.stdout:URL : http://www.gnu.org/software/grub/ 2020-03-11T22:29:09.608 INFO:teuthology.orchestra.run.smithi145.stdout:Summary : Support tools for GRUB. 2020-03-11T22:29:09.608 INFO:teuthology.orchestra.run.smithi145.stdout:Description : 2020-03-11T22:29:09.609 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:29:09.609 INFO:teuthology.orchestra.run.smithi145.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2020-03-11T22:29:09.609 INFO:teuthology.orchestra.run.smithi145.stdout:customizable bootloader with modular architecture. It supports a rich 2020-03-11T22:29:09.609 INFO:teuthology.orchestra.run.smithi145.stdout:variety of kernel formats, file systems, computer architectures and 2020-03-11T22:29:09.609 INFO:teuthology.orchestra.run.smithi145.stdout:hardware devices. 2020-03-11T22:29:09.609 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:29:09.609 INFO:teuthology.orchestra.run.smithi145.stdout:This subpackage provides tools for support of all platforms. 2020-03-11T22:29:09.610 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2020-03-11T22:29:09.610 INFO:teuthology.task.kernel:Updating grub on smithi145 to boot 4.18.0-147.el8.x86_64 2020-03-11T22:29:09.611 INFO:teuthology.orchestra.run.smithi145:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2020-03-11T22:29:10.506 INFO:teuthology.orchestra.run.smithi145.stderr:Generating grub configuration file ... 2020-03-11T22:29:12.523 INFO:teuthology.orchestra.run.smithi145.stderr:done 2020-03-11T22:29:12.525 INFO:teuthology.orchestra.run.smithi145:> mktemp 2020-03-11T22:29:12.556 INFO:teuthology.orchestra.run.smithi145.stdout:/tmp/tmp.ouxN54Drhm 2020-03-11T22:29:12.557 INFO:teuthology.orchestra.run.smithi145:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ouxN54Drhm 2020-03-11T22:29:12.642 INFO:teuthology.orchestra.run.smithi145:> sudo chmod 0666 /tmp/tmp.ouxN54Drhm 2020-03-11T22:29:12.779 DEBUG:teuthology.orchestra.remote:smithi145:/tmp/tmp.ouxN54Drhm is 4KB 2020-03-11T22:29:12.828 INFO:teuthology.orchestra.run.smithi145:> rm -fr /tmp/tmp.ouxN54Drhm 2020-03-11T22:29:12.842 INFO:teuthology.orchestra.run.smithi145:> sudo /bin/ls /boot/loader/entries || true 2020-03-11T22:29:12.910 INFO:teuthology.orchestra.run.smithi145.stdout:291b34331a614822a44a2b4be2021771-0-rescue.conf 2020-03-11T22:29:12.911 INFO:teuthology.orchestra.run.smithi145.stdout:291b34331a614822a44a2b4be2021771-4.18.0-147.5.1.el8_1.x86_64.conf 2020-03-11T22:29:12.911 INFO:teuthology.orchestra.run.smithi145.stdout:291b34331a614822a44a2b4be2021771-4.18.0-147.el8.x86_64.conf 2020-03-11T22:29:12.912 INFO:teuthology.orchestra.run.smithi145:> sudo grub2-set-default 291b34331a614822a44a2b4be2021771-4.18.0-147.el8.x86_64 2020-03-11T22:29:13.080 INFO:teuthology.orchestra.run.smithi145:> sudo shutdown -r now 2020-03-11T22:29:43.083 INFO:teuthology.misc:Re-opening connections... 2020-03-11T22:29:43.083 INFO:teuthology.misc:trying to connect to ubuntu@smithi145.front.sepia.ceph.com 2020-03-11T22:29:43.083 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:29:43.084 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi145.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:30:01.446 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.145 2020-03-11T22:30:01.446 DEBUG:teuthology.misc:waited 18.3630468845 2020-03-11T22:30:02.447 INFO:teuthology.misc:trying to connect to ubuntu@smithi145.front.sepia.ceph.com 2020-03-11T22:30:02.448 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:30:02.448 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi145.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:30:05.514 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.145 2020-03-11T22:30:05.514 DEBUG:teuthology.misc:waited 22.4307858944 2020-03-11T22:30:06.515 INFO:teuthology.misc:trying to connect to ubuntu@smithi145.front.sepia.ceph.com 2020-03-11T22:30:06.515 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:30:06.516 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi145.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:30:08.586 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.145 2020-03-11T22:30:08.586 DEBUG:teuthology.misc:waited 25.5028669834 2020-03-11T22:30:09.587 INFO:teuthology.misc:trying to connect to ubuntu@smithi145.front.sepia.ceph.com 2020-03-11T22:30:09.587 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:30:09.588 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi145.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:30:11.658 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.145 2020-03-11T22:30:11.658 DEBUG:teuthology.misc:waited 28.5748639107 2020-03-11T22:30:12.658 INFO:teuthology.misc:trying to connect to ubuntu@smithi145.front.sepia.ceph.com 2020-03-11T22:30:12.659 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:30:12.659 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi145.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:30:14.730 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.145 2020-03-11T22:30:14.730 DEBUG:teuthology.misc:waited 31.6469547749 2020-03-11T22:30:15.731 INFO:teuthology.misc:trying to connect to ubuntu@smithi145.front.sepia.ceph.com 2020-03-11T22:30:15.731 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:30:15.731 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi145.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:30:17.802 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.145 2020-03-11T22:30:17.802 DEBUG:teuthology.misc:waited 34.7188568115 2020-03-11T22:30:18.803 INFO:teuthology.misc:trying to connect to ubuntu@smithi145.front.sepia.ceph.com 2020-03-11T22:30:18.803 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:30:18.804 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi145.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:30:50.616 INFO:teuthology.orchestra.run.smithi145:> true 2020-03-11T22:30:50.958 DEBUG:teuthology.misc:waited 67.8744559288 2020-03-11T22:30:51.959 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2020-03-11T22:30:51.959 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-147.el8.x86_64"... 2020-03-11T22:30:51.959 INFO:teuthology.orchestra.run.smithi145:> uname -r 2020-03-11T22:30:51.976 INFO:teuthology.orchestra.run.smithi145.stdout:4.18.0-147.el8.x86_64 2020-03-11T22:30:51.976 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-147.el8.x86_64 vs 4.18.0-147.el8.x86_64 2020-03-11T22:30:51.976 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2020-03-11T22:30:52.977 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2020-03-11T22:30:52.977 INFO:teuthology.orchestra.run.smithi145:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-03-11T22:30:53.057 INFO:teuthology.orchestra.run.smithi145.stdout:ttyS1 2020-03-11T22:30:53.069 INFO:teuthology.run_tasks:Running task internal.base... 2020-03-11T22:30:53.077 INFO:teuthology.task.internal:Creating test directory... 2020-03-11T22:30:53.077 INFO:teuthology.orchestra.run.smithi145:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-03-11T22:30:53.092 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2020-03-11T22:30:53.104 INFO:teuthology.run_tasks:Running task internal.archive... 2020-03-11T22:30:53.116 INFO:teuthology.task.internal:Creating archive directory... 2020-03-11T22:30:53.116 INFO:teuthology.orchestra.run.smithi145:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-03-11T22:30:53.154 INFO:teuthology.run_tasks:Running task internal.coredump... 2020-03-11T22:30:53.170 INFO:teuthology.task.internal:Enabling coredump saving... 2020-03-11T22:30:53.171 INFO:teuthology.orchestra.run.smithi145:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-03-11T22:30:53.228 INFO:teuthology.orchestra.run.smithi145.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:30:53.241 INFO:teuthology.orchestra.run.smithi145.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:30:53.243 INFO:teuthology.run_tasks:Running task internal.sudo... 2020-03-11T22:30:53.256 INFO:teuthology.task.internal:Configuring sudo... 2020-03-11T22:30:53.256 INFO:teuthology.orchestra.run.smithi145:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-03-11T22:30:53.312 INFO:teuthology.run_tasks:Running task internal.syslog... 2020-03-11T22:30:53.331 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2020-03-11T22:30:53.332 INFO:teuthology.orchestra.run.smithi145:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-03-11T22:30:53.366 INFO:teuthology.orchestra.run.smithi145:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-03-11T22:30:53.439 INFO:teuthology.orchestra.run.smithi145:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2020-03-11T22:30:53.509 INFO:teuthology.orchestra.run.smithi145:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-03-11T22:30:53.539 INFO:teuthology.orchestra.run.smithi145:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2020-03-11T22:30:53.594 INFO:teuthology.orchestra.run.smithi145:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2020-03-11T22:30:53.678 INFO:teuthology.orchestra.run.smithi145:> sudo service rsyslog restart 2020-03-11T22:30:53.795 INFO:teuthology.orchestra.run.smithi145.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2020-03-11T22:30:53.929 INFO:teuthology.run_tasks:Running task internal.timer... 2020-03-11T22:30:53.939 INFO:teuthology.task.internal:Starting timer... 2020-03-11T22:30:53.939 INFO:teuthology.run_tasks:Running task pcp... 2020-03-11T22:30:54.015 INFO:teuthology.run_tasks:Running task selinux... 2020-03-11T22:30:54.029 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2020-03-11T22:30:54.029 INFO:teuthology.orchestra.run.smithi145:> sudo service auditd rotate 2020-03-11T22:30:54.093 INFO:teuthology.orchestra.run.smithi145.stdout:Rotating logs: [ OK ] 2020-03-11T22:30:54.095 DEBUG:teuthology.task.selinux:Getting current SELinux state 2020-03-11T22:30:54.095 INFO:teuthology.orchestra.run.smithi145:> /usr/sbin/getenforce 2020-03-11T22:30:54.129 INFO:teuthology.orchestra.run.smithi145.stdout:Permissive 2020-03-11T22:30:54.130 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi145.front.sepia.ceph.com': 'permissive'} 2020-03-11T22:30:54.130 INFO:teuthology.orchestra.run.smithi145:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|scontext=system_u:system_r:logrotate_t:s0\)' 2020-03-11T22:30:54.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-03-11T22:30:54.199 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2020-03-11T22:30:54.199 INFO:teuthology.orchestra.run.smithi145:> sudo /usr/sbin/setenforce permissive 2020-03-11T22:30:54.274 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2020-03-11T22:30:54.290 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated; assuming it is current 2020-03-11T22:30:54.290 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch origin/master 2020-03-11T22:30:54.483 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2020-03-11T22:30:54.487 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi145.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2020-03-11T22:30:55.275 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2020-03-11T22:30:57.303 INFO:teuthology.task.ansible.out: [WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2020-03-11T22:31:00.703 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2020-03-11T22:31:00.755 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:00 +0000 (0:00:00.153) 0:00:00.153 ******* 2020-03-11T22:31:04.486 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi145.front.sepia.ceph.com] 2020-03-11T22:31:04.563 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:04 +0000 (0:00:03.808) 0:00:03.961 ******* 2020-03-11T22:31:04.929 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2020-03-11T22:31:04.931 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-03-11T22:31:04.972 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:04 +0000 (0:00:00.408) 0:00:04.370 ******* 2020-03-11T22:31:05.540 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2020-03-11T22:31:05.542 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-03-11T22:31:05.588 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:05 +0000 (0:00:00.615) 0:00:04.986 ******* 2020-03-11T22:31:05.615 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2020-03-11T22:31:05.616 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:05.655 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:05 +0000 (0:00:00.066) 0:00:05.053 ******* 2020-03-11T22:31:05.986 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** ok: [smithi145.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-11T22:31:06.040 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:06 +0000 (0:00:00.384) 0:00:05.438 ******* 2020-03-11T22:31:07.088 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2020-03-11T22:31:07.090 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1583965866.06-187246913302575/source", "state": "file", "uid": 0} 2020-03-11T22:31:07.124 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:07 +0000 (0:00:01.084) 0:00:06.522 ******* 2020-03-11T22:31:07.588 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2020-03-11T22:31:07.589 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2020-03-11T22:31:07.590 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-03-11T22:31:07.688 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2020-03-11T22:31:07.694 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2020-03-11T22:31:07.887 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:07 +0000 (0:00:00.762) 0:00:07.285 ******* 2020-03-11T22:31:08.545 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi145.front.sepia.ceph.com] 2020-03-11T22:31:08.635 INFO:teuthology.task.ansible.out: [WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2020-03-11T22:31:08.637 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:08 +0000 (0:00:00.749) 0:00:08.034 ******* 2020-03-11T22:31:09.686 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2020-03-11T22:31:09.691 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.697529", "end": "2020-03-11 22:31:09.663879", "rc": 0, "start": "2020-03-11 22:31:08.966350", "stderr": "", "stderr_lines": [], "stdout": "d6cc316c8d11cadac2a03d4f84b215a35cf6c4b6", "stdout_lines": ["d6cc316c8d11cadac2a03d4f84b215a35cf6c4b6"]} 2020-03-11T22:31:09.727 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:09 +0000 (0:00:01.091) 0:00:09.125 ******* 2020-03-11T22:31:09.938 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2020-03-11T22:31:09.939 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.011555", "end": "2020-03-11 22:31:09.918129", "failed_when_result": false, "rc": 0, "start": "2020-03-11 22:31:09.906574", "stderr": "", "stderr_lines": [], "stdout": "d6cc316c8d11cadac2a03d4f84b215a35cf6c4b6", "stdout_lines": ["d6cc316c8d11cadac2a03d4f84b215a35cf6c4b6"]} 2020-03-11T22:31:09.988 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:09 +0000 (0:00:00.261) 0:00:09.386 ******* 2020-03-11T22:31:10.024 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2020-03-11T22:31:10.025 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"ansible_facts": {"perform_users_role": false}, "changed": false} 2020-03-11T22:31:10.060 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:10 +0000 (0:00:00.071) 0:00:09.458 ******* 2020-03-11T22:31:10.086 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2020-03-11T22:31:10.087 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.126 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:10 +0000 (0:00:00.065) 0:00:09.524 ******* 2020-03-11T22:31:10.151 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2020-03-11T22:31:10.153 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.189 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:10 +0000 (0:00:00.063) 0:00:09.587 ******* 2020-03-11T22:31:10.217 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2020-03-11T22:31:10.218 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.256 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:10 +0000 (0:00:00.066) 0:00:09.654 ******* 2020-03-11T22:31:10.292 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2020-03-11T22:31:10.293 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.330 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:10 +0000 (0:00:00.073) 0:00:09.728 ******* 2020-03-11T22:31:10.356 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2020-03-11T22:31:10.358 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.396 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:10 +0000 (0:00:00.066) 0:00:09.794 ******* 2020-03-11T22:31:10.453 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-03-11T22:31:10.454 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.465 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.476 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.477 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.488 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.499 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.499 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.510 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.521 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.534 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.536 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.546 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.546 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.557 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.558 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.569 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.580 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.580 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.592 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.592 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.603 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.604 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.615 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.626 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.626 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.648 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.648 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.659 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.660 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.671 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.671 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.682 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.683 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.694 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.704 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.716 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.716 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.727 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.738 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.749 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.750 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.761 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.772 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.772 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.774 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.784 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.795 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.796 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.806 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.817 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.818 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.829 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.840 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.840 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.851 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.862 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.873 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.874 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.885 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.896 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.897 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.908 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.919 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.930 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.932 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.946 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.965 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.966 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.977 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.978 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:10.990 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.001 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.001 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.012 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.012 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.023 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.024 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.036 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.037 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.048 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.060 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.071 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.083 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.096 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.107 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.118 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.119 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.131 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.132 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.133 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.145 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.147 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.158 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.159 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.161 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.171 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39', u'name': u'mgfritch'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.172 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d', u'name': u'xiubli'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.184 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f', u'name': u'jafaj'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.185 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef', u'name': u'jschmid'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.187 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518', u'name': u'kachau'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.197 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034', u'name': u'khiremat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.198 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'kapandya@thinkpad elavnUlTLKzuE7/eJ2nxMw 4ac31f4c197b2b6325a404950175faeedc105ea45092672e924b6edafe90af23', u'name': u'kapandya'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad elavnUlTLKzuE7/eJ2nxMw 4ac31f4c197b2b6325a404950175faeedc105ea45092672e924b6edafe90af23"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.258 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:11 +0000 (0:00:00.861) 0:00:10.656 ******* 2020-03-11T22:31:11.285 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-03-11T22:31:11.336 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:11 +0000 (0:00:00.077) 0:00:10.734 ******* 2020-03-11T22:31:11.367 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.410 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:11 +0000 (0:00:00.073) 0:00:10.808 ******* 2020-03-11T22:31:11.436 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2020-03-11T22:31:11.437 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.477 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:11 +0000 (0:00:00.067) 0:00:10.875 ******* 2020-03-11T22:31:11.503 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-03-11T22:31:11.505 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.542 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:11 +0000 (0:00:00.065) 0:00:10.940 ******* 2020-03-11T22:31:11.579 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-03-11T22:31:11.581 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.623 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:11 +0000 (0:00:00.080) 0:00:11.021 ******* 2020-03-11T22:31:11.649 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2020-03-11T22:31:11.650 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.690 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:11 +0000 (0:00:00.066) 0:00:11.088 ******* 2020-03-11T22:31:11.726 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-03-11T22:31:11.728 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.738 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.749 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.750 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.761 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.761 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.772 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.773 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.789 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.789 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.791 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.801 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.802 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.813 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.814 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.825 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.826 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.837 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.838 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.850 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.851 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.863 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.874 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.876 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.877 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.888 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.889 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.901 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.902 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.913 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.924 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.925 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.927 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.938 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.949 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.959 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.961 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.971 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.982 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.983 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:12.024 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:12 +0000 (0:00:00.334) 0:00:11.422 ******* 2020-03-11T22:31:12.785 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "9c97a4e7e2fa12022ca70c847609415d5b1cb1c9", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/keys-repo-sha1", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "state": "file", "uid": 0} 2020-03-11T22:31:12.823 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:12 +0000 (0:00:00.799) 0:00:12.221 ******* 2020-03-11T22:31:12.860 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2020-03-11T22:31:12.861 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => { "msg": "Host smithi145.front.sepia.ceph.com is running CentOS 8.1 (Core)" } 2020-03-11T22:31:12.903 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:12 +0000 (0:00:00.079) 0:00:12.301 ******* 2020-03-11T22:31:12.951 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2020-03-11T22:31:12.953 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2020-03-11T22:31:12.993 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:12 +0000 (0:00:00.088) 0:00:12.389 ******* 2020-03-11T22:31:13.019 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:13.067 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:13 +0000 (0:00:00.075) 0:00:12.465 ******* 2020-03-11T22:31:13.319 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-03-11T22:31:13.320 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.056843", "end": "2020-03-11 22:31:13.298878", "rc": 0, "start": "2020-03-11 22:31:13.242035", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-03-11T22:31:13.357 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:13 +0000 (0:00:00.290) 0:00:12.755 ******* 2020-03-11T22:31:13.393 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2020-03-11T22:31:13.395 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:13.436 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:13 +0000 (0:00:00.078) 0:00:12.834 ******* 2020-03-11T22:31:13.462 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-03-11T22:31:13.462 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:13.511 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:13 +0000 (0:00:00.074) 0:00:12.909 ******* 2020-03-11T22:31:13.537 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-03-11T22:31:13.539 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:13.575 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:13 +0000 (0:00:00.064) 0:00:12.973 ******* 2020-03-11T22:31:13.611 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-03-11T22:31:13.612 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:13.647 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:13 +0000 (0:00:00.072) 0:00:13.045 ******* 2020-03-11T22:31:13.673 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2020-03-11T22:31:13.675 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:13.711 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:13 +0000 (0:00:00.063) 0:00:13.109 ******* 2020-03-11T22:31:13.737 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2020-03-11T22:31:13.738 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:13.774 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:13 +0000 (0:00:00.062) 0:00:13.172 ******* 2020-03-11T22:31:13.810 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-03-11T22:31:13.812 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi145.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-11T22:31:13.847 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:13 +0000 (0:00:00.073) 0:00:13.245 ******* 2020-03-11T22:31:13.873 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2020-03-11T22:31:13.874 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:13.912 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:13 +0000 (0:00:00.064) 0:00:13.310 ******* 2020-03-11T22:31:13.938 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2020-03-11T22:31:13.939 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:13.975 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:13 +0000 (0:00:00.062) 0:00:13.373 ******* 2020-03-11T22:31:14.001 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-03-11T22:31:14.003 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-11T22:31:14.047 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:14 +0000 (0:00:00.071) 0:00:13.444 ******* 2020-03-11T22:31:14.072 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-03-11T22:31:14.073 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:14.112 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:14 +0000 (0:00:00.065) 0:00:13.510 ******* 2020-03-11T22:31:14.139 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2020-03-11T22:31:14.140 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-11T22:31:14.179 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:14 +0000 (0:00:00.066) 0:00:13.577 ******* 2020-03-11T22:31:14.208 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2020-03-11T22:31:14.209 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:14.263 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:14 +0000 (0:00:00.084) 0:00:13.661 ******* 2020-03-11T22:31:14.290 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-03-11T22:31:14.292 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:14.326 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:14 +0000 (0:00:00.062) 0:00:13.724 ******* 2020-03-11T22:31:14.362 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2020-03-11T22:31:14.364 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:14.403 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:14 +0000 (0:00:00.077) 0:00:13.801 ******* 2020-03-11T22:31:14.429 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-03-11T22:31:14.431 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:14.470 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:14 +0000 (0:00:00.066) 0:00:13.868 ******* 2020-03-11T22:31:14.506 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2020-03-11T22:31:14.507 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-11T22:31:14.554 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:14 +0000 (0:00:00.083) 0:00:13.952 ******* 2020-03-11T22:31:14.579 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-03-11T22:31:14.580 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:14.621 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:14 +0000 (0:00:00.067) 0:00:14.019 ******* 2020-03-11T22:31:14.655 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-03-11T22:31:14.657 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:14.696 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:14 +0000 (0:00:00.074) 0:00:14.094 ******* 2020-03-11T22:31:14.734 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2020-03-11T22:31:14.736 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:14.774 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:14 +0000 (0:00:00.078) 0:00:14.172 ******* 2020-03-11T22:31:14.810 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2020-03-11T22:31:14.812 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:14.861 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:14 +0000 (0:00:00.086) 0:00:14.259 ******* 2020-03-11T22:31:14.897 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2020-03-11T22:31:14.899 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:14.933 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:14 +0000 (0:00:00.071) 0:00:14.331 ******* 2020-03-11T22:31:14.959 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2020-03-11T22:31:14.961 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:14.998 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:14 +0000 (0:00:00.065) 0:00:14.396 ******* 2020-03-11T22:31:15.024 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2020-03-11T22:31:15.026 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-11T22:31:15.063 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:15 +0000 (0:00:00.065) 0:00:14.461 ******* 2020-03-11T22:31:15.092 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-03-11T22:31:15.131 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:15 +0000 (0:00:00.068) 0:00:14.529 ******* 2020-03-11T22:31:15.157 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2020-03-11T22:31:15.158 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:15.194 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:15 +0000 (0:00:00.062) 0:00:14.592 ******* 2020-03-11T22:31:15.417 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-03-11T22:31:15.419 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-11T22:31:15.455 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:15 +0000 (0:00:00.261) 0:00:14.853 ******* 2020-03-11T22:31:15.966 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-03-11T22:31:15.967 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/8/Everything/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2da7f101e29cfe8f3b630dbd6da82b4f9a749fee", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/8/Everything/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 304, "state": "file", "uid": 0} 2020-03-11T22:31:16.423 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/8/Everything/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "ddfaa38799b2dd4c23933a375aa4c3ff61bf7625", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/8/Everything/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 338, "state": "file", "uid": 0} 2020-03-11T22:31:16.461 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:16 +0000 (0:00:01.005) 0:00:15.859 ******* 2020-03-11T22:31:16.963 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-03-11T22:31:16.963 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2020-03-11T22:31:17.428 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2020-03-11T22:31:17.466 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:17 +0000 (0:00:01.004) 0:00:16.864 ******* 2020-03-11T22:31:17.503 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-03-11T22:31:17.505 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:17.542 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:17 +0000 (0:00:00.075) 0:00:16.940 ******* 2020-03-11T22:31:17.568 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-03-11T22:31:17.570 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:17.606 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:17 +0000 (0:00:00.063) 0:00:17.004 ******* 2020-03-11T22:31:17.631 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-03-11T22:31:17.633 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:17.670 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:17 +0000 (0:00:00.064) 0:00:17.068 ******* 2020-03-11T22:31:17.697 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-03-11T22:31:17.699 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:17.735 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:17 +0000 (0:00:00.064) 0:00:17.133 ******* 2020-03-11T22:31:17.760 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-03-11T22:31:17.761 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:17.798 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:17 +0000 (0:00:00.063) 0:00:17.196 ******* 2020-03-11T22:31:17.823 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-03-11T22:31:17.824 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:17.860 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:17 +0000 (0:00:00.061) 0:00:17.258 ******* 2020-03-11T22:31:17.886 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-03-11T22:31:17.888 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:17.925 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:17 +0000 (0:00:00.064) 0:00:17.323 ******* 2020-03-11T22:31:17.951 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-03-11T22:31:17.952 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:17.989 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:17 +0000 (0:00:00.063) 0:00:17.387 ******* 2020-03-11T22:31:18.015 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-03-11T22:31:18.016 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:18.052 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:18 +0000 (0:00:00.062) 0:00:17.450 ******* 2020-03-11T22:31:18.534 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2020-03-11T22:31:18.537 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.280575", "end": "2020-03-11 22:31:18.500307", "rc": 0, "start": "2020-03-11 22:31:18.219732", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:31:18.595 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:18 +0000 (0:00:00.543) 0:00:17.993 ******* 2020-03-11T22:31:21.835 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-03-11T22:31:21.836 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: krb5-workstation"]} 2020-03-11T22:31:21.873 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:21 +0000 (0:00:03.278) 0:00:21.271 ******* 2020-03-11T22:31:21.899 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-03-11T22:31:21.900 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:21.936 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:21 +0000 (0:00:00.062) 0:00:21.334 ******* 2020-03-11T22:31:21.961 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2020-03-11T22:31:21.963 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:21.997 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:21 +0000 (0:00:00.061) 0:00:21.395 ******* 2020-03-11T22:31:22.023 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-03-11T22:31:22.024 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:22.061 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:22 +0000 (0:00:00.063) 0:00:21.459 ******* 2020-03-11T22:31:22.086 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2020-03-11T22:31:22.088 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:22.122 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:22 +0000 (0:00:00.061) 0:00:21.520 ******* 2020-03-11T22:31:22.590 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2020-03-11T22:31:22.590 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2020-03-11T22:31:22.633 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:22 +0000 (0:00:00.510) 0:00:22.031 ******* 2020-03-11T22:31:23.554 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2020-03-11T22:31:23.554 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: smartmontools"]} 2020-03-11T22:31:23.602 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:23 +0000 (0:00:00.969) 0:00:23.000 ******* 2020-03-11T22:31:24.107 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-03-11T22:31:24.108 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2020-03-11T22:31:24.579 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2020-03-11T22:31:25.043 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2020-03-11T22:31:25.085 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:25 +0000 (0:00:01.481) 0:00:24.481 ******* 2020-03-11T22:31:25.350 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2020-03-11T22:31:25.350 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-11T22:31:25.390 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:25 +0000 (0:00:00.306) 0:00:24.788 ******* 2020-03-11T22:31:25.968 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-03-11T22:31:25.969 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 12725, "state": "file", "uid": 0} 2020-03-11T22:31:26.479 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2020-03-11T22:31:26.967 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2020-03-11T22:31:27.006 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:27 +0000 (0:00:01.615) 0:00:26.403 ******* 2020-03-11T22:31:27.041 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-03-11T22:31:27.043 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:27.085 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:27 +0000 (0:00:00.079) 0:00:26.483 ******* 2020-03-11T22:31:27.122 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-03-11T22:31:27.124 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2020-03-11T22:31:27.158 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:27 +0000 (0:00:00.072) 0:00:26.556 ******* 2020-03-11T22:31:27.369 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2020-03-11T22:31:27.370 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2020-03-11T22:31:27.405 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:27 +0000 (0:00:00.247) 0:00:26.803 ******* 2020-03-11T22:31:27.451 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2020-03-11T22:31:27.453 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:27.463 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:27.474 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:27.485 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:27.522 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:27 +0000 (0:00:00.116) 0:00:26.920 ******* 2020-03-11T22:31:28.464 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-03-11T22:31:28.465 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: nagios-common", "Installed: nrpe", "Installed: nagios-plugins", "Installed: nagios-plugins-load"]} 2020-03-11T22:31:28.504 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:28 +0000 (0:00:00.981) 0:00:27.902 ******* 2020-03-11T22:31:28.540 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2020-03-11T22:31:28.541 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:28.580 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:28 +0000 (0:00:00.076) 0:00:27.978 ******* 2020-03-11T22:31:29.075 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2020-03-11T22:31:29.075 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2020-03-11T22:31:29.114 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:29 +0000 (0:00:00.534) 0:00:28.512 ******* 2020-03-11T22:31:29.602 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-03-11T22:31:29.602 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2020-03-11T22:31:29.642 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:29 +0000 (0:00:00.527) 0:00:29.040 ******* 2020-03-11T22:31:29.678 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-03-11T22:31:29.679 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:29.716 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:29 +0000 (0:00:00.073) 0:00:29.114 ******* 2020-03-11T22:31:29.927 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-03-11T22:31:29.929 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-11T22:31:29.977 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:29 +0000 (0:00:00.261) 0:00:29.375 ******* 2020-03-11T22:31:30.220 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-03-11T22:31:30.222 INFO:teuthology.task.ansible.out:fatal: [smithi145.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2020-03-11T22:31:30.260 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:30 +0000 (0:00:00.282) 0:00:29.658 ******* 2020-03-11T22:31:30.296 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-03-11T22:31:30.298 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:30.335 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:30 +0000 (0:00:00.074) 0:00:29.733 ******* 2020-03-11T22:31:30.839 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2020-03-11T22:31:30.840 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "333141ba5e85aea94f51511a52d9b115d2e0112f", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1174, "state": "file", "uid": 0} 2020-03-11T22:31:30.882 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:30 +0000 (0:00:00.547) 0:00:30.280 ******* 2020-03-11T22:31:31.734 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-03-11T22:31:31.736 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2020-03-11 22:30:46 UTC", "ActiveEnterTimestampMonotonic": "11989383", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "network.target system.slice systemd-journald.socket var-run.mount tmp.mount time-sync.target systemd-tmpfiles-setup.service basic.target sysinit.target local-fs.target nss-lookup.target -.mount", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2020-03-11 22:30:46 UTC", "AssertTimestampMonotonic": "11988469", "Before": "multi-user.target shutdown.target plymouth-quit.service xdm.service getty@tty1.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2020-03-11 22:30:46 UTC", "ConditionTimestampMonotonic": "11988469", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "D 2020-03-11T22:31:31.738 INFO:teuthology.task.ansible.out:efaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "2388", "ExecMainStartTimestamp": "Wed 2020-03-11 22:30:46 UTC", "ExecMainStartTimestampMonotonic": "11989354", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f ; ignore_errors=no ; start_time=[Wed 2020-03-11 22:30:46 UTC] ; stop_time=[n/a] ; pid=2388 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "GID": "986", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2020-03-11 22:30:46 UTC", "InactiveExitTimestampMonotonic": "11989383", "InvocationID": "5b48ba86a5dd4ecba57d34adb3c6971c", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "Limi 2020-03-11T22:31:31.739 INFO:teuthology.task.ansible.out:tDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127436", "LimitNPROCSoft": "127436", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127436", "LimitSIGPENDINGSoft": "127436", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "2388", "MemoryAccounting": "yes", "MemoryCurrent": "1081344", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/run/nrpe /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "n 2020-03-11T22:31:31.740 INFO:teuthology.task.ansible.out:rpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2020-03-11 22:30:46 UTC", "StateChangeTimestampMonotonic": "11989383", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "990", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2020-03-11 22:30:46 UTC", "WatchdogTimestampMonotonic": "11989382", "WatchdogUSec": "0"}} 2020-03-11T22:31:31.776 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:31 +0000 (0:00:00.894) 0:00:31.174 ******* 2020-03-11T22:31:31.974 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2020-03-11T22:31:31.976 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001817", "end": "2020-03-11 22:31:31.953580", "rc": 0, "start": "2020-03-11 22:31:31.951763", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2020-03-11T22:31:32.012 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:32 +0000 (0:00:00.235) 0:00:31.410 ******* 2020-03-11T22:31:32.942 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2020-03-11T22:31:32.944 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: python3-libsemanage", "Installed: python3-policycoreutils"]} 2020-03-11T22:31:32.982 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:32 +0000 (0:00:00.970) 0:00:32.380 ******* 2020-03-11T22:31:33.905 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2020-03-11T22:31:33.906 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: selinux-policy-targeted"]} 2020-03-11T22:31:33.943 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:33 +0000 (0:00:00.960) 0:00:33.341 ******* 2020-03-11T22:31:35.369 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2020-03-11T22:31:35.410 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:35 +0000 (0:00:01.466) 0:00:34.808 ******* 2020-03-11T22:31:51.966 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2020-03-11T22:31:51.966 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:16.346326", "end": "2020-03-11 22:31:51.938840", "failed_when_result": false, "rc": 0, "start": "2020-03-11 22:31:35.592514", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2020-03-11T22:31:52.004 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:52 +0000 (0:00:16.594) 0:00:51.402 ******* 2020-03-11T22:31:52.496 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2020-03-11T22:31:52.496 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2020-03-11T22:31:52.535 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:52 +0000 (0:00:00.531) 0:00:51.933 ******* 2020-03-11T22:31:52.737 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2020-03-11T22:31:52.738 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.008679", "end": "2020-03-11 22:31:52.715461", "rc": 0, "start": "2020-03-11 22:31:52.706782", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:31:52.776 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:52 +0000 (0:00:00.240) 0:00:52.174 ******* 2020-03-11T22:31:52.971 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2020-03-11T22:31:52.972 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.002171", "end": "2020-03-11 22:31:52.951122", "rc": 0, "start": "2020-03-11 22:31:52.948951", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:31:53.026 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:53 +0000 (0:00:00.250) 0:00:52.424 ******* 2020-03-11T22:32:09.371 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2020-03-11T22:32:09.372 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:16.148012", "end": "2020-03-11 22:32:09.346121", "rc": 0, "start": "2020-03-11 22:31:53.198109", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:32:09.415 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:09 +0000 (0:00:16.388) 0:01:08.813 ******* 2020-03-11T22:32:09.626 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2020-03-11T22:32:09.628 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2020-03-11T22:32:09.673 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:09 +0000 (0:00:00.258) 0:01:09.071 ******* 2020-03-11T22:32:09.712 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2020-03-11T22:32:09.716 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi", "iozone"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2020-03-11T22:32:09.776 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:09 +0000 (0:00:00.102) 0:01:09.174 ******* 2020-03-11T22:32:09.824 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-03-11T22:32:09.826 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-03-11T22:32:09.864 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:09 +0000 (0:00:00.088) 0:01:09.262 ******* 2020-03-11T22:32:09.917 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2020-03-11T22:32:09.918 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ktdreyer/ceph-el8"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["@core", "@base", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2020-03-11T22:32:09.959 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:09 +0000 (0:00:00.094) 0:01:09.356 ******* 2020-03-11T22:32:10.005 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2020-03-11T22:32:10.006 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-03-11T22:32:10.045 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:10 +0000 (0:00:00.086) 0:01:09.443 ******* 2020-03-11T22:32:10.242 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2020-03-11T22:32:10.244 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-03-11T22:32:10.282 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:10 +0000 (0:00:00.236) 0:01:09.680 ******* 2020-03-11T22:32:10.473 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2020-03-11T22:32:10.475 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "gid": 1211, "name": "ubuntu", "state": "present", "system": false} 2020-03-11T22:32:10.512 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:10 +0000 (0:00:00.230) 0:01:09.910 ******* 2020-03-11T22:32:10.539 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-03-11T22:32:10.540 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.581 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:10 +0000 (0:00:00.068) 0:01:09.979 ******* 2020-03-11T22:32:10.608 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-03-11T22:32:10.610 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.654 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:10 +0000 (0:00:00.072) 0:01:10.052 ******* 2020-03-11T22:32:10.879 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2020-03-11T22:32:10.880 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2020-03-11T22:32:10.941 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:10 +0000 (0:00:00.287) 0:01:10.339 ******* 2020-03-11T22:32:11.525 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-03-11T22:32:11.526 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2020-03-11T22:32:11.571 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:11 +0000 (0:00:00.629) 0:01:10.969 ******* 2020-03-11T22:32:11.607 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-03-11T22:32:11.609 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi145"}, "changed": false} 2020-03-11T22:32:11.645 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:11 +0000 (0:00:00.073) 0:01:11.043 ******* 2020-03-11T22:32:12.671 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2020-03-11T22:32:12.672 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi145.front.sepia.ceph.com", "ansible_hostname": "smithi145", "ansible_nodename": "smithi145"}, "changed": true, "name": "smithi145"} 2020-03-11T22:32:12.709 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:12 +0000 (0:00:01.064) 0:01:12.107 ******* 2020-03-11T22:32:13.178 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2020-03-11T22:32:13.178 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2020-03-11T22:32:13.217 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:13 +0000 (0:00:00.507) 0:01:12.615 ******* 2020-03-11T22:32:13.688 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2020-03-11T22:32:13.689 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2020-03-11T22:32:13.726 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:13 +0000 (0:00:00.509) 0:01:13.124 ******* 2020-03-11T22:32:14.372 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2020-03-11T22:32:14.374 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"attempts": 1, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-03-11T22:32:14.418 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:14 +0000 (0:00:00.692) 0:01:13.816 ******* 2020-03-11T22:32:14.908 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2020-03-11T22:32:14.909 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2020-03-11T22:32:14.947 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:14 +0000 (0:00:00.528) 0:01:14.345 ******* 2020-03-11T22:32:15.148 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-03-11T22:32:15.149 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2020-03-11T22:32:15.186 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:15 +0000 (0:00:00.238) 0:01:14.584 ******* 2020-03-11T22:32:15.396 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-03-11T22:32:15.398 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2020-03-11T22:32:15.436 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:15 +0000 (0:00:00.250) 0:01:14.834 ******* 2020-03-11T22:32:15.664 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-03-11T22:32:15.666 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1211, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-03-11T22:32:15.712 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:15 +0000 (0:00:00.276) 0:01:15.110 ******* 2020-03-11T22:32:16.205 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-03-11T22:32:16.206 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2020-03-11T22:32:16.246 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:16 +0000 (0:00:00.533) 0:01:15.644 ******* 2020-03-11T22:32:16.733 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-03-11T22:32:16.734 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2020-03-11T22:32:16.772 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:16 +0000 (0:00:00.526) 0:01:16.170 ******* 2020-03-11T22:32:18.623 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-03-11T22:32:18.624 INFO:teuthology.task.ansible.out: [WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:01.647224", "end": "2020-03-11 22:32:18.592787", "rc": 0, "start": "2020-03-11 22:32:16.945563", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:32:18.660 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:18 +0000 (0:00:01.888) 0:01:18.058 ******* 2020-03-11T22:32:19.163 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2020-03-11T22:32:19.164 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "53b24162baf9c2fcf22da2b5c2c258a91997ffad", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 425, "state": "file", "uid": 0} 2020-03-11T22:32:19.612 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3e4763db3d6c8e2d723452e76e0b9ed9030ed28d", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 408, "state": "file", "uid": 0} 2020-03-11T22:32:20.065 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "6d4b75787c58f146a27a5ee2b688dd525283abf1", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 410, "state": "file", "uid": 0} 2020-03-11T22:32:20.106 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:20 +0000 (0:00:01.445) 0:01:19.504 ******* 2020-03-11T22:32:20.598 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-03-11T22:32:20.599 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'enabled': 1, u'name': u'lab-extras', u'baseurl': u'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/'}, 'key': u'lab-extras'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2020-03-11T22:32:20.639 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:20 +0000 (0:00:00.532) 0:01:20.037 ******* 2020-03-11T22:32:21.132 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-03-11T22:32:21.133 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 1, u'gpgkey': u'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial', u'enabled': 1, u'mirrorlist': u'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', u'name': u'CentOS-$releasever - Base'}, 'key': u'CentOS-Base'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2020-03-11T22:32:21.607 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 1, u'gpgkey': u'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial', u'enabled': 1, u'mirrorlist': u'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', u'name': u'CentOS-$releasever - AppStream'}, 'key': u'CentOS-AppStream'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2020-03-11T22:32:22.060 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 1, u'gpgkey': u'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial', u'enabled': 1, u'mirrorlist': u'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', u'name': u'CentOS-$releasever - Extras'}, 'key': u'CentOS-Extras'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2020-03-11T22:32:22.100 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:22 +0000 (0:00:01.461) 0:01:21.498 ******* 2020-03-11T22:32:22.640 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-03-11T22:32:22.641 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item=ktdreyer/ceph-el8) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ktdreyer/ceph-el8"], "delta": "0:00:00.328857", "end": "2020-03-11 22:32:22.611643", "item": "ktdreyer/ceph-el8", "rc": 0, "start": "2020-03-11 22:32:22.282786", "stderr": "", "stderr_lines": [], "stdout": "Repository successfully enabled.", "stdout_lines": ["Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} 2020-03-11T22:32:22.643 INFO:teuthology.task.ansible.out: [WARNING]: Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. 2020-03-11T22:32:22.686 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:22 +0000 (0:00:00.585) 0:01:22.083 ******* 2020-03-11T22:32:22.711 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2020-03-11T22:32:22.712 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:22.760 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:22 +0000 (0:00:00.074) 0:01:22.158 ******* 2020-03-11T22:32:22.798 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2020-03-11T22:32:22.800 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:22.811 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:22.851 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:22 +0000 (0:00:00.091) 0:01:22.249 ******* 2020-03-11T22:32:22.888 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2020-03-11T22:32:22.889 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:22.925 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:22 +0000 (0:00:00.073) 0:01:22.323 ******* 2020-03-11T22:32:22.960 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-11T22:32:22.962 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2020-03-11T22:32:23.000 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:22 +0000 (0:00:00.075) 0:01:22.398 ******* 2020-03-11T22:32:23.037 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:23.082 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:23 +0000 (0:00:00.081) 0:01:22.480 ******* 2020-03-11T22:32:23.121 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2020-03-11T22:32:23.123 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:23.161 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:23 +0000 (0:00:00.079) 0:01:22.559 ******* 2020-03-11T22:32:23.363 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2020-03-11T22:32:23.364 INFO:teuthology.task.ansible.out:fatal: [smithi145.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.006343", "end": "2020-03-11 22:32:23.343777", "msg": "non-zero return code", "rc": 1, "start": "2020-03-11 22:32:23.337434", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2020-03-11T22:32:23.403 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:23 +0000 (0:00:00.242) 0:01:22.801 ******* 2020-03-11T22:32:23.440 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-03-11T22:32:23.441 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:23.478 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:23 +0000 (0:00:00.074) 0:01:22.876 ******* 2020-03-11T22:32:25.947 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-03-11T22:32:25.947 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: qemu-kvm-block-rbd-15:2.12.0-88.module_el8.1.0+266+ba744077.2.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64"]} 2020-03-11T22:32:25.985 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:25 +0000 (0:00:02.506) 0:01:25.383 ******* 2020-03-11T22:32:28.968 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-03-11T22:32:28.968 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-6.el8.x86_64", "Removed: boost-random-1.66.0-6.el8.x86_64", "Removed: hdparm-9.54-2.el8.x86_64", "Removed: tuned-2.12.0-3.el8_1.1.noarch"]} 2020-03-11T22:32:29.007 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:29 +0000 (0:00:03.021) 0:01:28.405 ******* 2020-03-11T22:32:34.282 INFO:teuthology.task.ansible.out: 2020-03-11T22:32:34.283 INFO:teuthology.task.ansible.out:TASK [testnode : Base group workaround for BZ1782899] ************************** changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "cmd": "dnf -y groupinstall base --nobest && dnf -y group upgrade base", "delta": "0:00:05.048233", "end": "2020-03-11 22:32:34.259816", "rc": 0, "start": "2020-03-11 22:32:29.211583", "stderr": "No match for group package \"insights-client\"\nNo match for group package \"centos-release-eula\"\nNo match for group package \"centos-release-eula\"\nNo match for group package \"insights-client\"", "stderr_lines": ["No match for group package \"insights-client\"", "No match for group package \"centos-release-eula\"", "No match for group package \"centos-release-eula\"", "No match for group package \"insights-client\""], "stdout": "CentOS-8 - AppStream 142 kB/s | 4.3 kB 00:00 \nCentOS-8 - Base 130 kB/s | 3.8 kB 00:00 \nCentOS-8 - Extras 51 kB/s | 1.5 kB 00:00 \nCopr repo for ceph-el8 owned by ktdreyer 78 kB/s | 3.3 kB 00:00 \nExtra Packages for Enterprise Linux 166 kB/s | 5.3 kB 00:00 \nlab-extras 1.2 MB/s | 2.9 kB 00:00 \nDependencies resolved.\n================================================================================\n Package Architecture Version Repository Size\n================================================================================\nInstalling Groups:\n Base \n\nTransaction Summary\n================================================================================\n\nComplete!\nLast metadata expiration check: 0:00:03 ago on Wed 11 Mar 2020 10:32:29 PM UTC.\nDependencies resolved.\n================================================================================\n Package Architecture Version Repo 2020-03-11T22:32:34.285 INFO:teuthology.task.ansible.out:sitory Size\n================================================================================\nUpgrading Groups:\n Base \n\nTransaction Summary\n================================================================================\n\nComplete!", "stdout_lines": ["CentOS-8 - AppStream 142 kB/s | 4.3 kB 00:00 ", "CentOS-8 - Base 130 kB/s | 3.8 kB 00:00 ", "CentOS-8 - Extras 51 kB/s | 1.5 kB 00:00 ", "Copr repo for ceph-el8 owned by ktdreyer 78 kB/s | 3.3 kB 00:00 ", "Extra Packages for Enterprise Linux 166 kB/s | 5.3 kB 00:00 ", "lab-extras 1.2 MB/s | 2.9 kB 00:00 ", "Dependencies resolved.", "================================================================================", " Package Architecture Version Repository Size", "================================================================================", "Installing Groups:", " Base ", "", "Transaction Summary", "================================================================================", "", "Complete!", "Last metadata expiration check: 0:00:03 ago on Wed 11 Mar 2020 10:32:29 PM UTC.", "Dependencies resolved.", "================================================================================", " Package Architecture Version Repository Size", "================================================================================", "Upgrading Groups:", " Base ", "", "Transaction Summary", "================================================================================", "", "Complete!"]} 2020-03-11T22:32:34.322 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:34 +0000 (0:00:05.315) 0:01:33.720 ******* 2020-03-11T22:32:41.612 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-03-11T22:32:41.613 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Group core installed.", "Group base installed.", "Installed: dnf-utils", "Installed: sysstat", "Installed: libedit", "Installed: boost-thread", "Installed: xfsprogs", "Installed: gdisk", "Installed: parted", "Installed: libgcrypt", "Installed: fuse-libs", "Installed: openssl", "Installed: libuuid", "Installed: podman", "Installed: podman-docker", "Installed: attr", "Installed: ant", "Installed: lsof", "Installed: gettext", "Installed: bc", "Installed: xfsdump", "Installed: blktrace", "Installed: usbredir", "Installed: libev-devel", "Installed: valgrind", "Installed: nfs-utils", "Installed: ncurses-devel", "Installed: gcc", "Installed: git", "Installed: python3-nose", "Installed: python3-virtualenv", "Installed: genisoimage", "Installed: qemu-img", "Installed: qemu-kvm-core", "Installed: qemu-kvm-block-rbd", "Installed: libacl-devel", "Installed: dbench", "Installed: autoconf", "Installed: gdb", "Installed: tuned-2.12.0-3.el8_1.1.noarch", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: qemu-kvm-block-rbd-15:2.12.0-88.module_el8.1.0+266+ba744077.2.x86_64", "Installed: boost-program-options-1.66.0-6.el8.x86_64", "Installed: boost-random-1.66.0-6.el8.x86_64", "Installed: hdparm-9.54-2.el8.x86_64"]} 2020-03-11T22:32:41.659 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:41 +0000 (0:00:07.336) 0:01:41.056 ******* 2020-03-11T22:32:41.687 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-03-11T22:32:41.689 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:41.723 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:41 +0000 (0:00:00.064) 0:01:41.121 ******* 2020-03-11T22:32:42.647 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2020-03-11T22:32:42.685 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:42 +0000 (0:00:00.961) 0:01:42.083 ******* 2020-03-11T22:32:42.722 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-03-11T22:32:42.722 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:42.762 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:42 +0000 (0:00:00.077) 0:01:42.160 ******* 2020-03-11T22:32:43.090 INFO:teuthology.task.ansible.out: 2020-03-11T22:32:43.092 INFO:teuthology.task.ansible.out:TASK [testnode : Stop and disable firewalld] *********************************** ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "dbus.service polkit.service sysinit.target dbus.socket basic.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target network-pre.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_set 2020-03-11T22:32:43.094 INFO:teuthology.task.ansible.out:gid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "ip6tables.service ebtables.service shutdown.target ipset.service iptables.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonoton 2020-03-11T22:32:43.095 INFO:teuthology.task.ansible.out:ic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127436", "LimitNPROCSoft": "127436", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127436", "LimitSIGPENDINGSoft": "127436", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "Refu 2020-03-11T22:32:43.097 INFO:teuthology.task.ansible.out:seManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "dbus.socket sysinit.target system.slice", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2020-03-11T22:32:43.132 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:43 +0000 (0:00:00.370) 0:01:42.530 ******* 2020-03-11T22:32:43.134 INFO:teuthology.task.ansible.out: 2020-03-11T22:32:43.427 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2020-03-11T22:32:43.428 INFO:teuthology.task.ansible.out:fatal: [smithi145.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2020-03-11T22:32:43.464 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:43 +0000 (0:00:00.331) 0:01:42.862 ******* 2020-03-11T22:32:44.171 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2020-03-11T22:32:44.172 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2020-03-11T22:32:44.208 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:44 +0000 (0:00:00.744) 0:01:43.606 ******* 2020-03-11T22:32:44.234 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2020-03-11T22:32:44.235 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:44.271 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:44 +0000 (0:00:00.062) 0:01:43.669 ******* 2020-03-11T22:32:44.296 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-03-11T22:32:44.334 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:44 +0000 (0:00:00.063) 0:01:43.732 ******* 2020-03-11T22:32:44.359 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2020-03-11T22:32:44.361 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:44.396 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:44 +0000 (0:00:00.062) 0:01:43.794 ******* 2020-03-11T22:32:44.422 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2020-03-11T22:32:44.423 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:44.459 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:44 +0000 (0:00:00.062) 0:01:43.857 ******* 2020-03-11T22:32:44.498 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* skipping: [smithi145.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:44.535 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:44 +0000 (0:00:00.076) 0:01:43.933 ******* 2020-03-11T22:32:44.561 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-03-11T22:32:44.563 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:44.599 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:44 +0000 (0:00:00.063) 0:01:43.997 ******* 2020-03-11T22:32:44.627 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-03-11T22:32:44.670 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:44 +0000 (0:00:00.070) 0:01:44.068 ******* 2020-03-11T22:32:44.696 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2020-03-11T22:32:44.697 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:44.736 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:44 +0000 (0:00:00.066) 0:01:44.134 ******* 2020-03-11T22:32:44.762 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2020-03-11T22:32:44.764 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:44.799 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:44 +0000 (0:00:00.063) 0:01:44.197 ******* 2020-03-11T22:32:44.825 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2020-03-11T22:32:44.827 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:44.864 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:44 +0000 (0:00:00.064) 0:01:44.262 ******* 2020-03-11T22:32:44.889 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-03-11T22:32:44.891 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:44.927 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:44 +0000 (0:00:00.063) 0:01:44.325 ******* 2020-03-11T22:32:44.952 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2020-03-11T22:32:44.954 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:44.990 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:44 +0000 (0:00:00.063) 0:01:44.388 ******* 2020-03-11T22:32:45.016 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-03-11T22:32:45.018 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:45.052 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:45 +0000 (0:00:00.061) 0:01:44.450 ******* 2020-03-11T22:32:45.078 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2020-03-11T22:32:45.079 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:45.115 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:45 +0000 (0:00:00.063) 0:01:44.513 ******* 2020-03-11T22:32:45.141 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2020-03-11T22:32:45.143 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:45.178 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:45 +0000 (0:00:00.062) 0:01:44.576 ******* 2020-03-11T22:32:45.203 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-03-11T22:32:45.205 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:45.241 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:45 +0000 (0:00:00.062) 0:01:44.639 ******* 2020-03-11T22:32:45.267 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2020-03-11T22:32:45.268 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:45.303 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:45 +0000 (0:00:00.062) 0:01:44.701 ******* 2020-03-11T22:32:45.327 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2020-03-11T22:32:45.329 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:45.367 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:45 +0000 (0:00:00.063) 0:01:44.765 ******* 2020-03-11T22:32:45.392 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2020-03-11T22:32:45.394 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:45.429 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:45 +0000 (0:00:00.062) 0:01:44.827 ******* 2020-03-11T22:32:45.454 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2020-03-11T22:32:45.456 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:45.492 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:45 +0000 (0:00:00.063) 0:01:44.890 ******* 2020-03-11T22:32:45.518 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-03-11T22:32:45.520 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:45.557 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:45 +0000 (0:00:00.064) 0:01:44.955 ******* 2020-03-11T22:32:45.582 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-03-11T22:32:45.584 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:45.626 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:45 +0000 (0:00:00.069) 0:01:45.024 ******* 2020-03-11T22:32:45.651 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-03-11T22:32:45.653 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:45.687 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:45 +0000 (0:00:00.060) 0:01:45.085 ******* 2020-03-11T22:32:45.712 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-03-11T22:32:45.713 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:45.751 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:45 +0000 (0:00:00.063) 0:01:45.149 ******* 2020-03-11T22:32:45.775 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-03-11T22:32:45.776 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:45.811 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:45 +0000 (0:00:00.060) 0:01:45.209 ******* 2020-03-11T22:32:45.836 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-03-11T22:32:45.837 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:45.871 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:45 +0000 (0:00:00.059) 0:01:45.269 ******* 2020-03-11T22:32:45.895 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-03-11T22:32:45.896 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:45.929 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:45 +0000 (0:00:00.058) 0:01:45.327 ******* 2020-03-11T22:32:45.954 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-03-11T22:32:45.956 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:45.990 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:45 +0000 (0:00:00.060) 0:01:45.388 ******* 2020-03-11T22:32:46.015 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-03-11T22:32:46.017 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:46.051 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:46 +0000 (0:00:00.060) 0:01:45.449 ******* 2020-03-11T22:32:46.076 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-03-11T22:32:46.077 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:46.112 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:46 +0000 (0:00:00.060) 0:01:45.510 ******* 2020-03-11T22:32:46.136 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-03-11T22:32:46.138 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:46.174 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:46 +0000 (0:00:00.062) 0:01:45.572 ******* 2020-03-11T22:32:46.200 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-03-11T22:32:46.201 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:46.236 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:46 +0000 (0:00:00.061) 0:01:45.634 ******* 2020-03-11T22:32:46.260 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-03-11T22:32:46.262 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:46.295 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:46 +0000 (0:00:00.059) 0:01:45.693 ******* 2020-03-11T22:32:46.510 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-03-11T22:32:46.511 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2020-03-11T22:32:46.547 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:46 +0000 (0:00:00.251) 0:01:45.945 ******* 2020-03-11T22:32:46.757 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-03-11T22:32:46.759 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-11T22:32:46.794 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:46 +0000 (0:00:00.247) 0:01:46.192 ******* 2020-03-11T22:32:46.997 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-03-11T22:32:46.998 INFO:teuthology.task.ansible.out: [WARNING]: Cannot set fs attributes on a non-existent symlink target. follow should be set to False to avoid this. ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2020-03-11T22:32:47.035 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:47 +0000 (0:00:00.240) 0:01:46.433 ******* 2020-03-11T22:32:47.060 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-03-11T22:32:47.062 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:47.097 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:47 +0000 (0:00:00.061) 0:01:46.494 ******* 2020-03-11T22:32:47.121 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-03-11T22:32:47.123 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:47.156 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:47 +0000 (0:00:00.059) 0:01:46.554 ******* 2020-03-11T22:32:47.180 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-03-11T22:32:47.181 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:47.216 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:47 +0000 (0:00:00.060) 0:01:46.614 ******* 2020-03-11T22:32:47.242 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-03-11T22:32:47.243 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:47.277 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:47 +0000 (0:00:00.060) 0:01:46.675 ******* 2020-03-11T22:32:47.301 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-03-11T22:32:47.302 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:47.336 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:47 +0000 (0:00:00.058) 0:01:46.734 ******* 2020-03-11T22:32:47.360 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-03-11T22:32:47.362 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:47.397 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:47 +0000 (0:00:00.060) 0:01:46.795 ******* 2020-03-11T22:32:47.422 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-03-11T22:32:47.423 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:47.457 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:47 +0000 (0:00:00.060) 0:01:46.855 ******* 2020-03-11T22:32:47.482 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-03-11T22:32:47.484 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:47.518 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:47 +0000 (0:00:00.061) 0:01:46.916 ******* 2020-03-11T22:32:47.543 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-03-11T22:32:47.545 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:47.580 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:47 +0000 (0:00:00.061) 0:01:46.978 ******* 2020-03-11T22:32:47.606 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2020-03-11T22:32:47.607 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:47.643 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:47 +0000 (0:00:00.062) 0:01:47.041 ******* 2020-03-11T22:32:47.678 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-03-11T22:32:47.679 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:47.712 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:47 +0000 (0:00:00.069) 0:01:47.110 ******* 2020-03-11T22:32:47.738 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-03-11T22:32:47.739 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:47.776 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:47 +0000 (0:00:00.063) 0:01:47.174 ******* 2020-03-11T22:32:47.802 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-03-11T22:32:47.803 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:47.838 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:47 +0000 (0:00:00.062) 0:01:47.236 ******* 2020-03-11T22:32:47.864 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2020-03-11T22:32:47.865 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:47.900 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:47 +0000 (0:00:00.061) 0:01:47.298 ******* 2020-03-11T22:32:47.924 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2020-03-11T22:32:47.926 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:47.961 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:47 +0000 (0:00:00.060) 0:01:47.359 ******* 2020-03-11T22:32:47.986 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-03-11T22:32:47.988 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:48.030 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:48 +0000 (0:00:00.068) 0:01:47.428 ******* 2020-03-11T22:32:48.055 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-03-11T22:32:48.057 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:48.102 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:48 +0000 (0:00:00.072) 0:01:47.500 ******* 2020-03-11T22:32:48.128 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-03-11T22:32:48.130 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:48.167 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:48 +0000 (0:00:00.064) 0:01:47.565 ******* 2020-03-11T22:32:48.193 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-03-11T22:32:48.195 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:48.238 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:48 +0000 (0:00:00.071) 0:01:47.636 ******* 2020-03-11T22:32:48.264 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-03-11T22:32:48.265 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:48.309 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:48 +0000 (0:00:00.070) 0:01:47.707 ******* 2020-03-11T22:32:48.334 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-03-11T22:32:48.336 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:48.372 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:48 +0000 (0:00:00.062) 0:01:47.770 ******* 2020-03-11T22:32:48.396 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-03-11T22:32:48.398 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:48.439 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:48 +0000 (0:00:00.067) 0:01:47.837 ******* 2020-03-11T22:32:48.465 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-11T22:32:48.467 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-03-11T22:32:48.503 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:48 +0000 (0:00:00.064) 0:01:47.901 ******* 2020-03-11T22:32:48.539 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-03-11T22:32:48.541 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:48.550 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-03-11T22:32:48.561 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:48.561 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:48.572 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:48.583 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:48.584 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:48.628 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:48 +0000 (0:00:00.124) 0:01:48.026 ******* 2020-03-11T22:32:48.767 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-03-11T22:32:48.769 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003449", "end": "2020-03-11 22:32:48.750489", "rc": 0, "start": "2020-03-11 22:32:48.747040", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-03-11T22:32:48.807 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:48 +0000 (0:00:00.178) 0:01:48.205 ******* 2020-03-11T22:32:48.843 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-03-11T22:32:48.844 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:48.885 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:48 +0000 (0:00:00.078) 0:01:48.283 ******* 2020-03-11T22:32:48.911 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-03-11T22:32:48.912 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:48.947 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:48 +0000 (0:00:00.062) 0:01:48.345 ******* 2020-03-11T22:32:48.973 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-03-11T22:32:48.974 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:49.012 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:49 +0000 (0:00:00.064) 0:01:48.410 ******* 2020-03-11T22:32:49.037 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-03-11T22:32:49.039 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:49.074 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:49 +0000 (0:00:00.062) 0:01:48.472 ******* 2020-03-11T22:32:50.004 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2020-03-11T22:32:50.006 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: gdisk", "Installed: device-mapper"]} 2020-03-11T22:32:50.042 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:50 +0000 (0:00:00.968) 0:01:49.441 ******* 2020-03-11T22:32:50.078 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-03-11T22:32:50.079 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'block_used': 12964181, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 227130927, u'size_available': 930328276992, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60989857, u'device': u'/dev/sda1', u'inode_used': 65119, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227130927, "block_size": 4096, "block_total": 240095108, "block_used": 12964181, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989857, "inode_total": 61054976, "inode_used": 65119, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930328276992, "size_total": 983429562368, "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}} 2020-03-11T22:32:50.116 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:50 +0000 (0:00:00.073) 0:01:49.514 ******* 2020-03-11T22:32:50.316 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-03-11T22:32:50.318 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005065", "end": "2020-03-11 22:32:50.290170", "rc": 0, "start": "2020-03-11 22:32:50.285105", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:32:50.353 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:50 +0000 (0:00:00.236) 0:01:49.751 ******* 2020-03-11T22:32:50.400 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-03-11T22:32:50.401 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'block_used': 12964181, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 227130927, u'size_available': 930328276992, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60989857, u'device': u'/dev/sda1', u'inode_used': 65119, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227130927, "block_size": 4096, "block_total": 240095108, "block_used": 12964181, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989857, "inode_total": 61054976, "inode_used": 65119, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930328276992, "size_total": 983429562368, "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:50.440 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:50 +0000 (0:00:00.086) 0:01:49.838 ******* 2020-03-11T22:32:50.466 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-03-11T22:32:50.506 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:50 +0000 (0:00:00.063) 0:01:49.902 ******* 2020-03-11T22:32:50.666 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-03-11T22:32:50.667 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XLDP', u'wwn-0x5000c5009293fa4d'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5XLDP', u'wwn': u'0x5000c5009293fa4d', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'mq-deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XLDP-part1', u'wwn-0x5000c5009293fa4d-part1'], u'uuids': [u'ebbdc914-4a8c-428b-9507-ff6e0d466a46']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XLDP", "wwn-0x5000c5009293fa4d"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XLDP-part1", "wwn-0x5000c5009293fa4d-part1"], "labels": [], "masters": [], "uuids": ["ebbdc914-4a8c-428b-9507-ff6e0d466a46"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XLDP", "size": "931.51 GB", "support_discard": "0", "vend 2020-03-11T22:32:50.668 INFO:teuthology.task.ansible.out:or": "ATA", "virtual": 1, "wwn": "0x5000c5009293fa4d"}}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:52.203 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-ZKxc3d-VGQs-HeUq-70jI-g05e-WLTf-RHzclG', u'nvme-INTEL_SSDPEDMD400G4_PHFT620500N9400BGN', u'nvme-nvme.8086-504846543632303530304e3934303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.351022", "end": "2020-03-11 22:32:52.169533", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-ZKxc3d-VGQs-HeUq-70jI-g05e-WLTf-RHzclG", "nvme-INTEL_SSDPEDMD400G4_PHFT620500N9400BGN", "nvme-nvme.8086-504846543632303530304e3934303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:32:50.818511", "stderr": "", "stderr_lines": [], "stdout": "Creating new GP 2020-03-11T22:32:52.205 INFO:teuthology.task.ansible.out:T entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:32:53.470 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-4D1U1oDlK3PIkNf8KwgfPGmK2QPSN0adqT2T2UXUIi1DO7i8TCsNP61gqe2dqKFy'], u'uuids': [u'23a2ef35-9401-4164-8de8-e2f23313ca56']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.069364", "end": "2020-03-11 22:32:53.434385", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-4D1U1oDlK3PIkNf8KwgfPGmK2QPSN0adqT2T2UXUIi1DO7i8TCsNP61gqe2dqKFy"], "labels": [], "masters": [], "uuids": ["23a2ef35-9401-4164-8de8-e2f23313ca56"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:32:52.365021", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:32:54.712 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-4D1U1oDlK3PIkNf8KwgfPGmK2QPSN0adoDoZySbTDTTht1RB82r2AVDNJs36UQcl'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.048657", "end": "2020-03-11 22:32:54.677837", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-4D1U1oDlK3PIkNf8KwgfPGmK2QPSN0adoDoZySbTDTTht1RB82r2AVDNJs36UQcl"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:32:53.629180", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:32:55.959 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-4D1U1oDlK3PIkNf8KwgfPGmK2QPSN0ads3oZicxUHbZBwSC39huP7S3AkSmL5zJr'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.053728", "end": "2020-03-11 22:32:55.927775", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-4D1U1oDlK3PIkNf8KwgfPGmK2QPSN0ads3oZicxUHbZBwSC39huP7S3AkSmL5zJr"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:32:54.874047", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:32:57.208 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-4D1U1oDlK3PIkNf8KwgfPGmK2QPSN0aduo3Lo4YVj5hSQ2xA5GNwtyJr63bx3Uag'], u'uuids': [u'f466daaa-d522-4367-921f-64b73a76a0c5']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.052432", "end": "2020-03-11 22:32:57.170555", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-4D1U1oDlK3PIkNf8KwgfPGmK2QPSN0aduo3Lo4YVj5hSQ2xA5GNwtyJr63bx3Uag"], "labels": [], "masters": [], "uuids": ["f466daaa-d522-4367-921f-64b73a76a0c5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:32:56.118123", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:32:58.447 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-4D1U1oDlK3PIkNf8KwgfPGmK2QPSN0adPG95p13LNRDVqJjXSvMdElFSAO0A8oI7'], u'uuids': [u'bda90c32-45b7-4f38-bec0-f45dcff5fe1f']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.056564", "end": "2020-03-11 22:32:58.418315", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-4D1U1oDlK3PIkNf8KwgfPGmK2QPSN0adPG95p13LNRDVqJjXSvMdElFSAO0A8oI7"], "labels": [], "masters": [], "uuids": ["bda90c32-45b7-4f38-bec0-f45dcff5fe1f"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:32:57.361751", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:32:58.591 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:58 +0000 (0:00:08.086) 0:01:57.988 ******* 2020-03-11T22:32:58.628 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-03-11T22:32:58.629 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XLDP', u'wwn-0x5000c5009293fa4d'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5XLDP', u'wwn': u'0x5000c5009293fa4d', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'mq-deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XLDP-part1', u'wwn-0x5000c5009293fa4d-part1'], u'uuids': [u'ebbdc914-4a8c-428b-9507-ff6e0d466a46']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XLDP", "wwn-0x5000c5009293fa4d"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XLDP-part1", "wwn-0x5000c5009293fa4d-part1"], "labels": [], "masters": [], "uuids": ["ebbdc914-4a8c-428b-9507-ff6e0d466a46"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XLDP", "size": "931.51 GB", "support_discard": "0", "vend 2020-03-11T22:32:58.630 INFO:teuthology.task.ansible.out:or": "ATA", "virtual": 1, "wwn": "0x5000c5009293fa4d"}}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:58.868 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-ZKxc3d-VGQs-HeUq-70jI-g05e-WLTf-RHzclG', u'nvme-INTEL_SSDPEDMD400G4_PHFT620500N9400BGN', u'nvme-nvme.8086-504846543632303530304e3934303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.037639", "end": "2020-03-11 22:32:58.834987", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-ZKxc3d-VGQs-HeUq-70jI-g05e-WLTf-RHzclG", "nvme-INTEL_SSDPEDMD400G4_PHFT620500N9400BGN", "nvme-nvme.8086-504846543632303530304e3934303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:32:58.797348", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 11 2020-03-11T22:32:58.869 INFO:teuthology.task.ansible.out:0 MiB) copied, 0.0349674 s, 3.3 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0349674 s, 3.3 GB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:32:59.192 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-4D1U1oDlK3PIkNf8KwgfPGmK2QPSN0adqT2T2UXUIi1DO7i8TCsNP61gqe2dqKFy'], u'uuids': [u'23a2ef35-9401-4164-8de8-e2f23313ca56']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.138860", "end": "2020-03-11 22:32:59.158933", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-4D1U1oDlK3PIkNf8KwgfPGmK2QPSN0adqT2T2UXUIi1DO7i8TCsNP61gqe2dqKFy"], "labels": [], "masters": [], "uuids": ["23a2ef35-9401-4164-8de8-e2f23313ca56"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:32:59.020073", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.136323 s, 846 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.136323 s, 846 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:32:59.512 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-4D1U1oDlK3PIkNf8KwgfPGmK2QPSN0adoDoZySbTDTTht1RB82r2AVDNJs36UQcl'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.140483", "end": "2020-03-11 22:32:59.479934", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-4D1U1oDlK3PIkNf8KwgfPGmK2QPSN0adoDoZySbTDTTht1RB82r2AVDNJs36UQcl"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:32:59.339451", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.13789 s, 836 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.13789 s, 836 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:32:59.833 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-4D1U1oDlK3PIkNf8KwgfPGmK2QPSN0ads3oZicxUHbZBwSC39huP7S3AkSmL5zJr'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.146122", "end": "2020-03-11 22:32:59.806612", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-4D1U1oDlK3PIkNf8KwgfPGmK2QPSN0ads3oZicxUHbZBwSC39huP7S3AkSmL5zJr"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:32:59.660490", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.143621 s, 803 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.143621 s, 803 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:33:00.161 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-4D1U1oDlK3PIkNf8KwgfPGmK2QPSN0aduo3Lo4YVj5hSQ2xA5GNwtyJr63bx3Uag'], u'uuids': [u'f466daaa-d522-4367-921f-64b73a76a0c5']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.138568", "end": "2020-03-11 22:33:00.129506", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-4D1U1oDlK3PIkNf8KwgfPGmK2QPSN0aduo3Lo4YVj5hSQ2xA5GNwtyJr63bx3Uag"], "labels": [], "masters": [], "uuids": ["f466daaa-d522-4367-921f-64b73a76a0c5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:32:59.990938", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.136025 s, 848 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.136025 s, 848 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:33:00.479 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-4D1U1oDlK3PIkNf8KwgfPGmK2QPSN0adPG95p13LNRDVqJjXSvMdElFSAO0A8oI7'], u'uuids': [u'bda90c32-45b7-4f38-bec0-f45dcff5fe1f']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.139923", "end": "2020-03-11 22:33:00.451567", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-4D1U1oDlK3PIkNf8KwgfPGmK2QPSN0adPG95p13LNRDVqJjXSvMdElFSAO0A8oI7"], "labels": [], "masters": [], "uuids": ["bda90c32-45b7-4f38-bec0-f45dcff5fe1f"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:33:00.311644", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.137207 s, 841 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.137207 s, 841 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:33:00.529 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:00 +0000 (0:00:01.938) 0:01:59.927 ******* 2020-03-11T22:33:00.794 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-03-11T22:33:00.795 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.064007", "end": "2020-03-11 22:33:00.760395", "rc": 0, "start": "2020-03-11 22:33:00.696388", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:33:00.831 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:00 +0000 (0:00:00.301) 0:02:00.229 ******* 2020-03-11T22:33:01.095 INFO:teuthology.task.ansible.out: 2020-03-11T22:33:01.097 INFO:teuthology.task.ansible.out:TASK [testnode : Check for physical volumes] *********************************** changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.067591", "end": "2020-03-11 22:33:01.068498", "rc": 0, "start": "2020-03-11 22:33:01.000907", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:33:01.136 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:01 +0000 (0:00:00.304) 0:02:00.534 ******* 2020-03-11T22:33:01.162 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-03-11T22:33:01.204 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:01 +0000 (0:00:00.067) 0:02:00.602 ******* 2020-03-11T22:33:01.229 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-03-11T22:33:01.231 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:01.271 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:01 +0000 (0:00:00.067) 0:02:00.669 ******* 2020-03-11T22:33:01.297 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-03-11T22:33:01.298 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:01.340 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:01 +0000 (0:00:00.069) 0:02:00.738 ******* 2020-03-11T22:33:01.365 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-03-11T22:33:01.367 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:01.406 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:01 +0000 (0:00:00.066) 0:02:00.804 ******* 2020-03-11T22:33:01.432 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-03-11T22:33:01.433 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:01.469 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:01 +0000 (0:00:00.062) 0:02:00.867 ******* 2020-03-11T22:33:01.660 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-03-11T22:33:01.660 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-11T22:33:01.700 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:01 +0000 (0:00:00.230) 0:02:01.097 ******* 2020-03-11T22:33:01.747 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-03-11T22:33:01.747 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'block_used': 12964181, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 227130927, u'size_available': 930328276992, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60989857, u'device': u'/dev/sda1', u'inode_used': 65119, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227130927, "block_size": 4096, "block_total": 240095108, "block_used": 12964181, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989857, "inode_total": 61054976, "inode_used": 65119, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930328276992, "size_total": 983429562368, "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:01.785 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:01 +0000 (0:00:00.085) 0:02:01.183 ******* 2020-03-11T22:33:01.810 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2020-03-11T22:33:01.811 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:01.847 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:01 +0000 (0:00:00.062) 0:02:01.245 ******* 2020-03-11T22:33:01.883 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-11T22:33:01.885 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:01.920 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:01 +0000 (0:00:00.072) 0:02:01.318 ******* 2020-03-11T22:33:01.948 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-11T22:33:01.950 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:01.987 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:01 +0000 (0:00:00.067) 0:02:01.385 ******* 2020-03-11T22:33:02.013 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-03-11T22:33:02.014 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:02.050 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:02 +0000 (0:00:00.062) 0:02:01.448 ******* 2020-03-11T22:33:02.076 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:02.113 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:02 +0000 (0:00:00.063) 0:02:01.511 ******* 2020-03-11T22:33:02.138 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-03-11T22:33:02.140 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:02.175 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:02 +0000 (0:00:00.062) 0:02:01.573 ******* 2020-03-11T22:33:02.745 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-03-11T22:33:02.746 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'pvs': u'/dev/nvme0n1'}, 'key': u'vg_nvme'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-03-11T22:33:02.786 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:02 +0000 (0:00:00.610) 0:02:02.184 ******* 2020-03-11T22:33:03.313 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-03-11T22:33:03.315 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-11T22:33:03.634 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-11T22:33:03.968 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-11T22:33:04.314 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-11T22:33:04.646 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-11T22:33:04.715 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:04 +0000 (0:00:01.929) 0:02:04.113 ******* 2020-03-11T22:33:04.911 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-03-11T22:33:04.952 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:04 +0000 (0:00:00.237) 0:02:04.350 ******* 2020-03-11T22:33:04.998 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-03-11T22:33:05.000 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:05.188 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-11T22:33:05.377 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-11T22:33:05.553 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-11T22:33:05.751 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-11T22:33:05.796 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:05 +0000 (0:00:00.844) 0:02:05.194 ******* 2020-03-11T22:33:06.012 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2020-03-11T22:33:06.012 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-11T22:33:06.050 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:06 +0000 (0:00:00.252) 0:02:05.447 ******* 2020-03-11T22:33:06.086 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2020-03-11T22:33:06.088 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-03-11T22:33:06.128 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:06 +0000 (0:00:00.078) 0:02:05.526 ******* 2020-03-11T22:33:07.565 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2020-03-11T22:33:07.567 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true} 2020-03-11T22:33:07.614 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:07 +0000 (0:00:01.477) 0:02:07.004 ******* 2020-03-11T22:33:08.118 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2020-03-11T22:33:08.120 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-03-11T22:33:08.159 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:08 +0000 (0:00:00.553) 0:02:07.557 ******* 2020-03-11T22:33:08.185 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:08.228 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:08 +0000 (0:00:00.068) 0:02:07.626 ******* 2020-03-11T22:33:08.254 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2020-03-11T22:33:08.256 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:08.292 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:08 +0000 (0:00:00.063) 0:02:07.690 ******* 2020-03-11T22:33:08.319 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2020-03-11T22:33:08.321 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:08.356 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:08 +0000 (0:00:00.064) 0:02:07.754 ******* 2020-03-11T22:33:08.382 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2020-03-11T22:33:08.383 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:08.419 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:08 +0000 (0:00:00.062) 0:02:07.817 ******* 2020-03-11T22:33:08.445 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2020-03-11T22:33:08.446 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:08.481 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:08 +0000 (0:00:00.062) 0:02:07.879 ******* 2020-03-11T22:33:08.510 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2020-03-11T22:33:08.511 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:08.547 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:08 +0000 (0:00:00.065) 0:02:07.945 ******* 2020-03-11T22:33:08.573 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2020-03-11T22:33:08.574 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:08.610 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:08 +0000 (0:00:00.062) 0:02:08.008 ******* 2020-03-11T22:33:08.635 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2020-03-11T22:33:08.637 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:08.674 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:08 +0000 (0:00:00.063) 0:02:08.072 ******* 2020-03-11T22:33:08.701 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-03-11T22:33:08.702 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:08.740 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:08 +0000 (0:00:00.066) 0:02:08.138 ******* 2020-03-11T22:33:08.766 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-03-11T22:33:08.768 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:08.822 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:08 +0000 (0:00:00.082) 0:02:08.220 ******* 2020-03-11T22:33:08.849 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-03-11T22:33:08.851 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:08.886 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:08 +0000 (0:00:00.063) 0:02:08.284 ******* 2020-03-11T22:33:09.377 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-03-11T22:33:09.378 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2020-03-11T22:33:09.416 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:09 +0000 (0:00:00.529) 0:02:08.814 ******* 2020-03-11T22:33:09.728 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-03-11T22:33:09.730 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2020-03-11 22:30:46 UTC", "ActiveEnterTimestampMonotonic": "11569407", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-tmpfiles-setup.service basic.target sntp.service system.slice tmp.mount ntpdate.service sysinit.target ntpd.service -.mount systemd-journald.socket", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2020-03-11 22:30:45 UTC", "AssertTimestampMonotonic": "10412747", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2020-03-11 22:30:45 UTC", "ConditionTimestampMonotonic": "10412709", "ConfigurationDirectoryMode": "0755", "Conflicts": "ntpd.service systemd-timesyncd.service shutdown.target", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description" 2020-03-11T22:33:09.732 INFO:teuthology.task.ansible.out:: "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "2318", "ExecMainStartTimestamp": "Wed 2020-03-11 22:30:45 UTC", "ExecMainStartTimestampMonotonic": "10504644", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2020-03-11 22:30:45 UTC", "InactiveExitTimestampMonotonic": "10438563", "InvocationID": "e72da479a7144c82aff8df9545bf595c", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "167772 2020-03-11T22:33:09.733 INFO:teuthology.task.ansible.out:16", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127436", "LimitNPROCSoft": "127436", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127436", "LimitSIGPENDINGSoft": "127436", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "2318", "MemoryAccounting": "yes", "MemoryCurrent": "2891776", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardE 2020-03-11T22:33:09.735 INFO:teuthology.task.ansible.out:rror": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2020-03-11 22:30:46 UTC", "StateChangeTimestampMonotonic": "11569407", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2020-03-11 22:30:45 UTC", "WatchdogTimestampMonotonic": "10504653", "WatchdogUSec": "0"}} 2020-03-11T22:33:09.769 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:09 +0000 (0:00:00.352) 0:02:09.167 ******* 2020-03-11T22:33:09.794 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-03-11T22:33:09.795 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:09.830 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:09 +0000 (0:00:00.061) 0:02:09.228 ******* 2020-03-11T22:33:09.855 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-03-11T22:33:09.857 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:09.892 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:09 +0000 (0:00:00.062) 0:02:09.290 ******* 2020-03-11T22:33:09.917 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2020-03-11T22:33:09.919 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:09.953 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:09 +0000 (0:00:00.061) 0:02:09.351 ******* 2020-03-11T22:33:10.154 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-03-11T22:33:10.155 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "gid": 1211, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2020-03-11T22:33:10.191 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:10 +0000 (0:00:00.238) 0:02:09.589 ******* 2020-03-11T22:33:10.683 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2020-03-11T22:33:10.683 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1211, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2020-03-11T22:33:10.720 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:10 +0000 (0:00:00.528) 0:02:10.118 ******* 2020-03-11T22:33:11.900 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2020-03-11T22:33:11.901 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-03-11T22:33:11.949 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:11 +0000 (0:00:01.229) 0:02:11.347 ******* 2020-03-11T22:33:11.976 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2020-03-11T22:33:11.977 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:12.018 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:12 +0000 (0:00:00.069) 0:02:11.417 ******* 2020-03-11T22:33:12.046 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2020-03-11T22:33:12.047 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2020-03-11T22:33:12.084 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:12 +0000 (0:00:00.065) 0:02:11.482 ******* 2020-03-11T22:33:12.279 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2020-03-11T22:33:12.281 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2020-03-11T22:33:12.317 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:12 +0000 (0:00:00.232) 0:02:11.715 ******* 2020-03-11T22:33:12.342 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2020-03-11T22:33:12.344 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:12.380 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:12 +0000 (0:00:00.063) 0:02:11.778 ******* 2020-03-11T22:33:12.407 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2020-03-11T22:33:12.409 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-03-11T22:33:12.443 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:12 +0000 (0:00:00.062) 0:02:11.841 ******* 2020-03-11T22:33:12.469 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2020-03-11T22:33:12.470 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:12.520 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:12 +0000 (0:00:00.077) 0:02:11.918 ******* 2020-03-11T22:33:12.714 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2020-03-11T22:33:12.716 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-03-11T22:33:12.750 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:12 +0000 (0:00:00.230) 0:02:12.148 ******* 2020-03-11T22:33:12.943 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2020-03-11T22:33:12.945 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-03-11T22:33:12.981 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:12 +0000 (0:00:00.230) 0:02:12.379 ******* 2020-03-11T22:33:13.010 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2020-03-11T22:33:13.011 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:13.052 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:13 +0000 (0:00:00.070) 0:02:12.450 ******* 2020-03-11T22:33:13.088 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2020-03-11T22:33:13.089 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2020-03-11T22:33:13.100 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2020-03-11T22:33:13.101 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2020-03-11T22:33:13.112 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2020-03-11T22:33:13.155 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:13 +0000 (0:00:00.102) 0:02:12.553 ******* 2020-03-11T22:33:13.191 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2020-03-11T22:33:13.192 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'lo', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:13.203 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno2', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:13.204 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:13.216 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f0', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:13.254 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:13 +0000 (0:00:00.099) 0:02:12.652 ******* 2020-03-11T22:33:13.280 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-11T22:33:13.281 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:13.319 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:13 +0000 (0:00:00.064) 0:02:12.717 ******* 2020-03-11T22:33:13.346 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2020-03-11T22:33:13.348 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:13.384 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:13 +0000 (0:00:00.064) 0:02:12.782 ******* 2020-03-11T22:33:13.410 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2020-03-11T22:33:13.411 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:13.453 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:13 +0000 (0:00:00.068) 0:02:12.851 ******* 2020-03-11T22:33:13.479 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2020-03-11T22:33:13.480 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:13.516 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:13 +0000 (0:00:00.063) 0:02:12.914 ******* 2020-03-11T22:33:13.542 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-03-11T22:33:13.543 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:13.593 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:13 +0000 (0:00:00.076) 0:02:12.991 ******* 2020-03-11T22:33:13.619 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-03-11T22:33:13.620 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:13.657 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:13 +0000 (0:00:00.064) 0:02:13.055 ******* 2020-03-11T22:33:13.683 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2020-03-11T22:33:13.685 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:13.722 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:13 +0000 (0:00:00.064) 0:02:13.120 ******* 2020-03-11T22:33:13.748 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-03-11T22:33:13.749 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:13.788 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:13 +0000 (0:00:00.066) 0:02:13.186 ******* 2020-03-11T22:33:15.019 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-03-11T22:33:15.020 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:01.006046", "end": "2020-03-11 22:33:14.977569", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-03-11 22:33:13.971523", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:33:15.205 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003042", "end": "2020-03-11 22:33:15.173074", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-03-11 22:33:15.170032", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:33:15.390 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002862", "end": "2020-03-11 22:33:15.355572", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-03-11 22:33:15.352710", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2020-03-11T22:33:15.612 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.030164", "end": "2020-03-11 22:33:15.576225", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-03-11 22:33:15.546061", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:33:15.653 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:15 +0000 (0:00:01.864) 0:02:15.051 ******* 2020-03-11T22:33:15.679 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-03-11T22:33:15.728 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:15 +0000 (0:00:00.074) 0:02:15.126 ******* 2020-03-11T22:33:15.921 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2020-03-11T22:33:15.921 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2020-03-11T22:33:16.028 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2020-03-11T22:33:16.031 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2020-03-11T22:33:16.033 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched 2020-03-11T22:33:16.034 INFO:teuthology.task.ansible.out: PLAY RECAP ********************************************************************* smithi145.front.sepia.ceph.com : ok=111 changed=33 unreachable=0 failed=0 skipped=171 rescued=0 ignored=3 Wednesday 11 March 2020 22:33:16 +0000 (0:00:00.305) 0:02:15.431 ******* =============================================================================== common : nrpe - Remove SELinux policy package -------------------------- 16.59s common : nrpe - Load SELinux policy package ---------------------------- 16.39s testnode : Zap all non-root disks --------------------------------------- 8.09s 2020-03-11T22:33:16.034 INFO:teuthology.task.ansible.out:testnode : Install packages --------------------------------------------- 7.34s testnode : Base group workaround for BZ1782899 -------------------------- 5.32s Gathering Facts --------------------------------------------------------- 3.81s common : Install Kerberos Packages (RedHat) ----------------------------- 3.28s testnode : Ensure ceph dependency packages are not present. ------------- 3.02s testnode : Ensure ceph packages are not present. ------------------------ 2.51s testnode : Blow away lingering OSD data and FSIDs ----------------------- 1.94s testnode : Create logical volume(s) ------------------------------------- 1.93s 2020-03-11T22:33:16.034 INFO:teuthology.task.ansible.out:testnode : Rebuild rpmdb ------------------------------------------------ 1.89s testnode : List any leftover Ceph artifacts from previous jobs ---------- 1.86s common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/. --- 1.62s common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/. --- 1.48s testnode : Create xfs filesystem on /dev/vg_nvme/lv_5 ------------------- 1.48s common : nrpe - Set SELinux boolean nagios_run_sudo true ---------------- 1.47s testnode : Configure version specific repos in /etc/yum.repos.d/ -------- 1.46s testnode : Configure local mirrorlists ---------------------------------- 1.45s 2020-03-11T22:33:16.035 INFO:teuthology.task.ansible.out:testnode : Install packages via pip --------------------- 2020-03-11T22:33:16.036 INFO:teuthology.task.ansible.out:---------------- 1.23s 2020-03-11T22:33:16.334 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi145.front.sepia.ceph.com')] 2020-03-11T22:33:16.337 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:33:16.337 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi145.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:33:16.409 INFO:teuthology.orchestra.run.smithi145:> true 2020-03-11T22:33:16.493 INFO:teuthology.run_tasks:Running task clock... 2020-03-11T22:33:16.505 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2020-03-11T22:33:16.506 INFO:teuthology.orchestra.run.smithi145:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-03-11T22:33:16.567 INFO:teuthology.orchestra.run.smithi145.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2020-03-11T22:33:16.585 INFO:teuthology.orchestra.run.smithi145.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2020-03-11T22:33:16.627 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: ntpd: command not found 2020-03-11T22:33:16.641 INFO:teuthology.orchestra.run.smithi145.stdout:506 Cannot talk to daemon 2020-03-11T22:33:16.658 INFO:teuthology.orchestra.run.smithi145.stderr:Failed to start ntp.service: Unit ntp.service not found. 2020-03-11T22:33:16.676 INFO:teuthology.orchestra.run.smithi145.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2020-03-11T22:33:16.740 INFO:teuthology.orchestra.run.smithi145.stderr:bash: ntpq: command not found 2020-03-11T22:33:16.742 INFO:teuthology.orchestra.run.smithi145.stdout:210 Number of sources = 4 2020-03-11T22:33:16.742 INFO:teuthology.orchestra.run.smithi145.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2020-03-11T22:33:16.743 INFO:teuthology.orchestra.run.smithi145.stdout:=============================================================================== 2020-03-11T22:33:16.743 INFO:teuthology.orchestra.run.smithi145.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-11T22:33:16.743 INFO:teuthology.orchestra.run.smithi145.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-11T22:33:16.743 INFO:teuthology.orchestra.run.smithi145.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-11T22:33:16.743 INFO:teuthology.orchestra.run.smithi145.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-11T22:33:16.743 INFO:teuthology.run_tasks:Running task install... 2020-03-11T22:33:16.755 DEBUG:teuthology.task.install:project ceph 2020-03-11T22:33:16.755 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '4acc1121e47039a3cdf2978ae458c10d153aa141'}} 2020-03-11T22:33:16.755 DEBUG:teuthology.task.install:config {'sha1': '4acc1121e47039a3cdf2978ae458c10d153aa141'} 2020-03-11T22:33:16.755 INFO:teuthology.task.install:Using flavor: basic 2020-03-11T22:33:16.767 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2020-03-11T22:33:16.767 INFO:teuthology.task.install:extra packages: [] 2020-03-11T22:33:16.768 INFO:teuthology.orchestra.run.smithi145:> sudo lsb_release -is 2020-03-11T22:33:16.818 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS 2020-03-11T22:33:16.819 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-11T22:33:16.820 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'sha1': '4acc1121e47039a3cdf2978ae458c10d153aa141', 'exclude_packages': [], 'debuginfo': None, 'install_ceph_packages': True, 'wait_for_package': False, 'downgrade_packages': [], 'project': 'ceph', 'tag': None, 'extra_packages': [], 'branch': None, 'packages': {}, 'repos_only': False, 'extra_system_packages': [], 'flavor': 'basic', 'extras': None} 2020-03-11T22:33:16.820 INFO:teuthology.orchestra.run.smithi145:> sudo lsb_release -is 2020-03-11T22:33:16.895 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS 2020-03-11T22:33:16.896 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-11T22:33:16.896 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-11T22:33:17.116 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-zafman-octo-diag/4acc1121e47039a3cdf2978ae458c10d153aa141/centos/8/flavors/default/ 2020-03-11T22:33:17.117 INFO:teuthology.task.install.rpm:Package version is 15.1.0-1941.g4acc112 2020-03-11T22:33:17.297 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-zafman-octo-diag/4acc1121e47039a3cdf2978ae458c10d153aa141/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-zafman-octo-diag/4acc1121e47039a3cdf2978ae458c10d153aa141/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-zafman-octo-diag/4acc1121e47039a3cdf2978ae458c10d153aa141/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2020-03-11T22:33:17.297 INFO:teuthology.orchestra.run.smithi145:> sudo sh -c 'cat > /etc/yum.repos.d/ceph.repo' 2020-03-11T22:33:17.342 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2020-03-11T22:33:17.343 INFO:teuthology.orchestra.run.smithi145:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4acc1121e47039a3cdf2978ae458c10d153aa141/;g' /etc/yum.repos.d/ceph.repo ; fi 2020-03-11T22:33:17.411 INFO:teuthology.orchestra.run.smithi145:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2020-03-11T22:33:17.492 INFO:teuthology.orchestra.run.smithi145:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2020-03-11T22:33:17.559 INFO:teuthology.orchestra.run.smithi145.stdout:check_obsoletes = 1 2020-03-11T22:33:17.560 INFO:teuthology.orchestra.run.smithi145:> sudo yum clean all 2020-03-11T22:33:17.942 INFO:teuthology.orchestra.run.smithi145.stdout:239 files removed 2020-03-11T22:33:17.971 INFO:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-radosgw 2020-03-11T22:33:18.642 INFO:teuthology.orchestra.run.smithi145.stdout:ceph packages for x86_64 208 kB/s | 69 kB 00:00 2020-03-11T22:33:18.939 INFO:teuthology.orchestra.run.smithi145.stdout:ceph noarch packages 50 kB/s | 14 kB 00:00 2020-03-11T22:33:19.266 INFO:teuthology.orchestra.run.smithi145.stdout:ceph source packages 4.6 kB/s | 1.4 kB 00:00 2020-03-11T22:33:19.426 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS-8 - AppStream 47 MB/s | 6.5 MB 00:00 2020-03-11T22:33:20.557 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS-8 - Base 42 MB/s | 5.0 MB 00:00 2020-03-11T22:33:21.348 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS-8 - Extras 33 kB/s | 2.1 kB 00:00 2020-03-11T22:33:21.490 INFO:teuthology.orchestra.run.smithi145.stdout:Copr repo for ceph-el8 owned by ktdreyer 390 kB/s | 38 kB 00:00 2020-03-11T22:33:21.856 INFO:teuthology.orchestra.run.smithi145.stdout:Extra Packages for Enterprise Linux 19 MB/s | 6.0 MB 00:00 2020-03-11T22:33:23.094 INFO:teuthology.orchestra.run.smithi145.stdout:lab-extras 5.7 MB/s | 30 kB 00:00 2020-03-11T22:33:24.062 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2020-03-11T22:33:24.065 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:33:24.065 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2020-03-11T22:33:24.065 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:33:24.065 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2020-03-11T22:33:24.065 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-radosgw x86_64 2:15.1.0-1941.g4acc112.el8 ceph 9.8 M 2020-03-11T22:33:24.066 INFO:teuthology.orchestra.run.smithi145.stdout:Upgrading: 2020-03-11T22:33:24.066 INFO:teuthology.orchestra.run.smithi145.stdout: librados2 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 3.6 M 2020-03-11T22:33:24.066 INFO:teuthology.orchestra.run.smithi145.stdout: librbd1 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 1.9 M 2020-03-11T22:33:24.066 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2020-03-11T22:33:24.066 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-base x86_64 2:15.1.0-1941.g4acc112.el8 ceph 5.5 M 2020-03-11T22:33:24.066 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-common x86_64 2:15.1.0-1941.g4acc112.el8 ceph 21 M 2020-03-11T22:33:24.066 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-selinux x86_64 2:15.1.0-1941.g4acc112.el8 ceph 25 k 2020-03-11T22:33:24.067 INFO:teuthology.orchestra.run.smithi145.stdout: libcephfs2 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 507 k 2020-03-11T22:33:24.067 INFO:teuthology.orchestra.run.smithi145.stdout: libradosstriper1 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 377 k 2020-03-11T22:33:24.067 INFO:teuthology.orchestra.run.smithi145.stdout: librgw2 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 6.2 M 2020-03-11T22:33:24.067 INFO:teuthology.orchestra.run.smithi145.stdout: python3-ceph-argparse x86_64 2:15.1.0-1941.g4acc112.el8 ceph 40 k 2020-03-11T22:33:24.067 INFO:teuthology.orchestra.run.smithi145.stdout: python3-ceph-common x86_64 2:15.1.0-1941.g4acc112.el8 ceph 49 k 2020-03-11T22:33:24.067 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cephfs x86_64 2:15.1.0-1941.g4acc112.el8 ceph 162 k 2020-03-11T22:33:24.067 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rados x86_64 2:15.1.0-1941.g4acc112.el8 ceph 352 k 2020-03-11T22:33:24.067 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rbd x86_64 2:15.1.0-1941.g4acc112.el8 ceph 309 k 2020-03-11T22:33:24.068 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rgw x86_64 2:15.1.0-1941.g4acc112.el8 ceph 113 k 2020-03-11T22:33:24.068 INFO:teuthology.orchestra.run.smithi145.stdout: librdkafka x86_64 0.11.4-1.el8 CentOS-AppStream 353 k 2020-03-11T22:33:24.068 INFO:teuthology.orchestra.run.smithi145.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2020-03-11T22:33:24.068 INFO:teuthology.orchestra.run.smithi145.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2020-03-11T22:33:24.068 INFO:teuthology.orchestra.run.smithi145.stdout: librabbitmq x86_64 0.9.0-1.el8 CentOS-Base 47 k 2020-03-11T22:33:24.068 INFO:teuthology.orchestra.run.smithi145.stdout: gperftools-libs x86_64 2.7-6.el8 epel 315 k 2020-03-11T22:33:24.068 INFO:teuthology.orchestra.run.smithi145.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2020-03-11T22:33:24.069 INFO:teuthology.orchestra.run.smithi145.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2020-03-11T22:33:24.069 INFO:teuthology.orchestra.run.smithi145.stdout: leveldb x86_64 1.21-2.el8 lab-extras 179 k 2020-03-11T22:33:24.069 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:33:24.069 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2020-03-11T22:33:24.069 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:33:24.069 INFO:teuthology.orchestra.run.smithi145.stdout:Install 21 Packages 2020-03-11T22:33:24.069 INFO:teuthology.orchestra.run.smithi145.stdout:Upgrade 2 Packages 2020-03-11T22:33:24.070 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:33:24.070 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 51 M 2020-03-11T22:33:24.070 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2020-03-11T22:33:25.667 INFO:teuthology.orchestra.run.smithi145.stdout:(1/23): ceph-base-15.1.0-1941.g4acc112.el8.x86_ 3.4 MB/s | 5.5 MB 00:01 2020-03-11T22:33:25.701 INFO:teuthology.orchestra.run.smithi145.stdout:(2/23): ceph-selinux-15.1.0-1941.g4acc112.el8.x 737 kB/s | 25 kB 00:00 2020-03-11T22:33:25.776 INFO:teuthology.orchestra.run.smithi145.stdout:(3/23): libcephfs2-15.1.0-1941.g4acc112.el8.x86 6.6 MB/s | 507 kB 00:00 2020-03-11T22:33:25.845 INFO:teuthology.orchestra.run.smithi145.stdout:(4/23): libradosstriper1-15.1.0-1941.g4acc112.e 5.3 MB/s | 377 kB 00:00 2020-03-11T22:33:26.067 INFO:teuthology.orchestra.run.smithi145.stdout:(5/23): ceph-radosgw-15.1.0-1941.g4acc112.el8.x 4.9 MB/s | 9.8 MB 00:01 2020-03-11T22:33:26.119 INFO:teuthology.orchestra.run.smithi145.stdout:(6/23): python3-ceph-argparse-15.1.0-1941.g4acc 764 kB/s | 40 kB 00:00 2020-03-11T22:33:26.159 INFO:teuthology.orchestra.run.smithi145.stdout:(7/23): python3-ceph-common-15.1.0-1941.g4acc11 1.2 MB/s | 49 kB 00:00 2020-03-11T22:33:26.204 INFO:teuthology.orchestra.run.smithi145.stdout:(8/23): python3-cephfs-15.1.0-1941.g4acc112.el8 3.5 MB/s | 162 kB 00:00 2020-03-11T22:33:26.259 INFO:teuthology.orchestra.run.smithi145.stdout:(9/23): python3-rados-15.1.0-1941.g4acc112.el8. 6.3 MB/s | 352 kB 00:00 2020-03-11T22:33:26.315 INFO:teuthology.orchestra.run.smithi145.stdout:(10/23): python3-rbd-15.1.0-1941.g4acc112.el8.x 5.4 MB/s | 309 kB 00:00 2020-03-11T22:33:26.365 INFO:teuthology.orchestra.run.smithi145.stdout:(11/23): python3-rgw-15.1.0-1941.g4acc112.el8.x 2.2 MB/s | 113 kB 00:00 2020-03-11T22:33:26.399 INFO:teuthology.orchestra.run.smithi145.stdout:(12/23): librdkafka-0.11.4-1.el8.x86_64.rpm 10 MB/s | 353 kB 00:00 2020-03-11T22:33:26.403 INFO:teuthology.orchestra.run.smithi145.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 65 MB/s | 259 kB 00:00 2020-03-11T22:33:26.405 INFO:teuthology.orchestra.run.smithi145.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 31 MB/s | 44 kB 00:00 2020-03-11T22:33:26.406 INFO:teuthology.orchestra.run.smithi145.stdout:(15/23): librabbitmq-0.9.0-1.el8.x86_64.rpm 31 MB/s | 47 kB 00:00 2020-03-11T22:33:26.469 INFO:teuthology.orchestra.run.smithi145.stdout:(16/23): librgw2-15.1.0-1941.g4acc112.el8.x86_6 10 MB/s | 6.2 MB 00:00 2020-03-11T22:33:26.499 INFO:teuthology.orchestra.run.smithi145.stdout:(17/23): gperftools-libs-2.7-6.el8.x86_64.rpm 3.3 MB/s | 315 kB 00:00 2020-03-11T22:33:26.517 INFO:teuthology.orchestra.run.smithi145.stdout:(18/23): libunwind-1.3.1-3.el8.x86_64.rpm 4.6 MB/s | 75 kB 00:00 2020-03-11T22:33:26.520 INFO:teuthology.orchestra.run.smithi145.stdout:(19/23): leveldb-1.21-2.el8.x86_64.rpm 38 MB/s | 179 kB 00:00 2020-03-11T22:33:26.530 INFO:teuthology.orchestra.run.smithi145.stdout:(20/23): liboath-2.6.2-3.el8.x86_64.rpm 958 kB/s | 59 kB 00:00 2020-03-11T22:33:26.867 INFO:teuthology.orchestra.run.smithi145.stdout:(21/23): ceph-common-15.1.0-1941.g4acc112.el8.x 7.4 MB/s | 21 MB 00:02 2020-03-11T22:33:26.875 INFO:teuthology.orchestra.run.smithi145.stdout:(22/23): librbd1-15.1.0-1941.g4acc112.el8.x86_6 5.5 MB/s | 1.9 MB 00:00 2020-03-11T22:33:26.887 INFO:teuthology.orchestra.run.smithi145.stdout:(23/23): librados2-15.1.0-1941.g4acc112.el8.x86 9.7 MB/s | 3.6 MB 00:00 2020-03-11T22:33:26.890 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:33:26.890 INFO:teuthology.orchestra.run.smithi145.stdout:Total 18 MB/s | 51 MB 00:02 2020-03-11T22:33:26.895 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2020-03-11T22:33:26.938 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2020-03-11T22:33:26.938 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2020-03-11T22:33:27.304 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2020-03-11T22:33:27.304 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2020-03-11T22:33:28.315 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2020-03-11T22:33:28.514 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2020-03-11T22:33:28.686 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2020-03-11T22:33:29.363 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2020-03-11T22:33:29.511 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : librados2-2:15.1.0-1941.g4acc112.el8.x86_64 3/25 2020-03-11T22:33:29.590 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: librados2-2:15.1.0-1941.g4acc112.el8.x86_64 3/25 2020-03-11T22:33:29.770 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-rados-2:15.1.0-1941.g4acc112.el8.x86_64 4/25 2020-03-11T22:33:29.936 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 5/25 2020-03-11T22:33:30.160 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 5/25 2020-03-11T22:33:30.344 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 6/25 2020-03-11T22:33:30.408 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 6/25 2020-03-11T22:33:30.606 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : librabbitmq-0.9.0-1.el8.x86_64 7/25 2020-03-11T22:33:30.761 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 8/25 2020-03-11T22:33:31.382 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 8/25 2020-03-11T22:33:31.544 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : librgw2-2:15.1.0-1941.g4acc112.el8.x86_64 9/25 2020-03-11T22:33:31.610 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: librgw2-2:15.1.0-1941.g4acc112.el8.x86_64 9/25 2020-03-11T22:33:31.745 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-ceph-argparse-2:15.1.0-1941.g4acc112.el8.x 10/25 2020-03-11T22:33:31.897 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-cephfs-2:15.1.0-1941.g4acc112.el8.x86_64 11/25 2020-03-11T22:33:32.140 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-rgw-2:15.1.0-1941.g4acc112.el8.x86_64 12/25 2020-03-11T22:33:32.361 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-rbd-2:15.1.0-1941.g4acc112.el8.x86_64 13/25 2020-03-11T22:33:32.502 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libradosstriper1-2:15.1.0-1941.g4acc112.el8.x86_64 14/25 2020-03-11T22:33:32.576 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: libradosstriper1-2:15.1.0-1941.g4acc112.el8.x86_64 14/25 2020-03-11T22:33:32.773 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/25 2020-03-11T22:33:32.962 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : gperftools-libs-2.7-6.el8.x86_64 16/25 2020-03-11T22:33:33.135 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : leveldb-1.21-2.el8.x86_64 17/25 2020-03-11T22:33:33.309 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2020-03-11T22:33:33.486 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-ceph-common-2:15.1.0-1941.g4acc112.el8.x86 19/25 2020-03-11T22:33:36.914 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 20/25 2020-03-11T22:33:37.119 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 20/25 2020-03-11T22:33:37.781 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 20/25 2020-03-11T22:33:38.019 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 21/25 2020-03-11T22:33:38.108 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 21/25 2020-03-11T22:33:38.251 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-selinux-2:15.1.0-1941.g4acc112.el8.x86_64 22/25 2020-03-11T22:33:54.557 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-selinux-2:15.1.0-1941.g4acc112.el8.x86_64 22/25 2020-03-11T22:33:54.557 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /sys 2020-03-11T22:33:54.557 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /proc 2020-03-11T22:33:54.557 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /dev 2020-03-11T22:33:54.558 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /run 2020-03-11T22:33:54.558 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /mnt 2020-03-11T22:33:54.558 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /var/tmp 2020-03-11T22:33:54.558 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /home 2020-03-11T22:33:54.558 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /tmp 2020-03-11T22:33:54.558 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /dev 2020-03-11T22:33:54.558 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:33:55.582 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-radosgw-2:15.1.0-1941.g4acc112.el8.x86_64 23/25 2020-03-11T22:33:55.659 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-radosgw-2:15.1.0-1941.g4acc112.el8.x86_64 23/25 2020-03-11T22:33:55.659 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2020-03-11T22:33:55.825 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2020-03-11T22:33:55.825 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2020-03-11T22:33:56.493 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2020-03-11T22:33:56.493 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 1/25 2020-03-11T22:33:56.493 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 2/25 2020-03-11T22:33:56.494 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-radosgw-2:15.1.0-1941.g4acc112.el8.x86_64 3/25 2020-03-11T22:33:56.494 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-selinux-2:15.1.0-1941.g4acc112.el8.x86_64 4/25 2020-03-11T22:33:56.494 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 5/25 2020-03-11T22:33:56.494 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libradosstriper1-2:15.1.0-1941.g4acc112.el8.x86_64 6/25 2020-03-11T22:33:56.494 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librgw2-2:15.1.0-1941.g4acc112.el8.x86_64 7/25 2020-03-11T22:33:56.494 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-ceph-argparse-2:15.1.0-1941.g4acc112.el8.x 8/25 2020-03-11T22:33:56.494 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-ceph-common-2:15.1.0-1941.g4acc112.el8.x86 9/25 2020-03-11T22:33:56.495 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-cephfs-2:15.1.0-1941.g4acc112.el8.x86_64 10/25 2020-03-11T22:33:56.495 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-rados-2:15.1.0-1941.g4acc112.el8.x86_64 11/25 2020-03-11T22:33:56.495 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-rbd-2:15.1.0-1941.g4acc112.el8.x86_64 12/25 2020-03-11T22:33:56.495 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-rgw-2:15.1.0-1941.g4acc112.el8.x86_64 13/25 2020-03-11T22:33:56.495 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 14/25 2020-03-11T22:33:56.495 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2020-03-11T22:33:56.495 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2020-03-11T22:33:56.495 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librabbitmq-0.9.0-1.el8.x86_64 17/25 2020-03-11T22:33:56.496 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : gperftools-libs-2.7-6.el8.x86_64 18/25 2020-03-11T22:33:56.496 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2020-03-11T22:33:56.496 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2020-03-11T22:33:56.496 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : leveldb-1.21-2.el8.x86_64 21/25 2020-03-11T22:33:56.496 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librados2-2:15.1.0-1941.g4acc112.el8.x86_64 22/25 2020-03-11T22:33:56.496 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2020-03-11T22:33:56.496 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 24/25 2020-03-11T22:33:57.137 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2020-03-11T22:33:57.138 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:33:57.138 INFO:teuthology.orchestra.run.smithi145.stdout:Upgraded: 2020-03-11T22:33:57.138 INFO:teuthology.orchestra.run.smithi145.stdout: librados2-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:33:57.138 INFO:teuthology.orchestra.run.smithi145.stdout: librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:33:57.138 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:33:57.138 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2020-03-11T22:33:57.138 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-radosgw-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:33:57.139 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:33:57.139 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:33:57.139 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-selinux-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:33:57.139 INFO:teuthology.orchestra.run.smithi145.stdout: libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:33:57.139 INFO:teuthology.orchestra.run.smithi145.stdout: libradosstriper1-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:33:57.139 INFO:teuthology.orchestra.run.smithi145.stdout: librgw2-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:33:57.139 INFO:teuthology.orchestra.run.smithi145.stdout: python3-ceph-argparse-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:33:57.140 INFO:teuthology.orchestra.run.smithi145.stdout: python3-ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:33:57.140 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cephfs-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:33:57.140 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rados-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:33:57.140 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rbd-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:33:57.140 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rgw-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:33:57.140 INFO:teuthology.orchestra.run.smithi145.stdout: librdkafka-0.11.4-1.el8.x86_64 2020-03-11T22:33:57.140 INFO:teuthology.orchestra.run.smithi145.stdout: lttng-ust-2.8.1-11.el8.x86_64 2020-03-11T22:33:57.140 INFO:teuthology.orchestra.run.smithi145.stdout: python3-prettytable-0.7.2-14.el8.noarch 2020-03-11T22:33:57.141 INFO:teuthology.orchestra.run.smithi145.stdout: librabbitmq-0.9.0-1.el8.x86_64 2020-03-11T22:33:57.141 INFO:teuthology.orchestra.run.smithi145.stdout: gperftools-libs-2.7-6.el8.x86_64 2020-03-11T22:33:57.141 INFO:teuthology.orchestra.run.smithi145.stdout: liboath-2.6.2-3.el8.x86_64 2020-03-11T22:33:57.141 INFO:teuthology.orchestra.run.smithi145.stdout: libunwind-1.3.1-3.el8.x86_64 2020-03-11T22:33:57.141 INFO:teuthology.orchestra.run.smithi145.stdout: leveldb-1.21-2.el8.x86_64 2020-03-11T22:33:57.141 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:33:57.142 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2020-03-11T22:33:57.406 INFO:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-test 2020-03-11T22:33:57.815 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:00:34 ago on Wed 11 Mar 2020 10:33:23 PM UTC. 2020-03-11T22:33:58.208 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2020-03-11T22:33:58.209 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:33:58.209 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2020-03-11T22:33:58.209 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:33:58.209 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2020-03-11T22:33:58.209 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-test x86_64 2:15.1.0-1941.g4acc112.el8 ceph 42 M 2020-03-11T22:33:58.209 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2020-03-11T22:33:58.210 INFO:teuthology.orchestra.run.smithi145.stdout: jq x86_64 1.5-12.el8 CentOS-AppStream 161 k 2020-03-11T22:33:58.210 INFO:teuthology.orchestra.run.smithi145.stdout: oniguruma x86_64 6.8.2-1.el8 CentOS-AppStream 188 k 2020-03-11T22:33:58.210 INFO:teuthology.orchestra.run.smithi145.stdout: socat x86_64 1.7.3.2-6.el8 CentOS-AppStream 298 k 2020-03-11T22:33:58.210 INFO:teuthology.orchestra.run.smithi145.stdout: libxslt x86_64 1.1.32-3.el8 CentOS-Base 249 k 2020-03-11T22:33:58.210 INFO:teuthology.orchestra.run.smithi145.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2020-03-11T22:33:58.210 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:33:58.210 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2020-03-11T22:33:58.211 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:33:58.211 INFO:teuthology.orchestra.run.smithi145.stdout:Install 6 Packages 2020-03-11T22:33:58.211 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:33:58.211 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 43 M 2020-03-11T22:33:58.212 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 184 M 2020-03-11T22:33:58.212 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2020-03-11T22:33:58.243 INFO:teuthology.orchestra.run.smithi145.stdout:(1/6): jq-1.5-12.el8.x86_64.rpm 4.9 MB/s | 161 kB 00:00 2020-03-11T22:33:58.254 INFO:teuthology.orchestra.run.smithi145.stdout:(2/6): oniguruma-6.8.2-1.el8.x86_64.rpm 4.3 MB/s | 188 kB 00:00 2020-03-11T22:33:58.255 INFO:teuthology.orchestra.run.smithi145.stdout:(3/6): socat-1.7.3.2-6.el8.x86_64.rpm 43 MB/s | 298 kB 00:00 2020-03-11T22:33:58.258 INFO:teuthology.orchestra.run.smithi145.stdout:(4/6): libxslt-1.1.32-3.el8.x86_64.rpm 62 MB/s | 249 kB 00:00 2020-03-11T22:33:58.318 INFO:teuthology.orchestra.run.smithi145.stdout:(5/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 1.1 MB/s | 69 kB 00:00 2020-03-11T22:34:01.115 INFO:teuthology.orchestra.run.smithi145.stdout:(6/6): ceph-test-15.1.0-1941.g4acc112.el8.x86_6 14 MB/s | 42 MB 00:02 2020-03-11T22:34:01.118 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:34:01.118 INFO:teuthology.orchestra.run.smithi145.stdout:Total 15 MB/s | 43 MB 00:02 2020-03-11T22:34:01.124 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2020-03-11T22:34:01.146 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2020-03-11T22:34:01.147 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2020-03-11T22:34:01.406 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2020-03-11T22:34:01.406 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2020-03-11T22:34:01.844 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2020-03-11T22:34:02.035 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libxslt-1.1.32-3.el8.x86_64 1/6 2020-03-11T22:34:02.213 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 2/6 2020-03-11T22:34:02.431 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : socat-1.7.3.2-6.el8.x86_64 3/6 2020-03-11T22:34:02.572 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : oniguruma-6.8.2-1.el8.x86_64 4/6 2020-03-11T22:34:02.645 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: oniguruma-6.8.2-1.el8.x86_64 4/6 2020-03-11T22:34:06.646 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : jq-1.5-12.el8.x86_64 5/6 2020-03-11T22:34:06.835 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-test-2:15.1.0-1941.g4acc112.el8.x86_64 6/6 2020-03-11T22:34:07.026 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-test-2:15.1.0-1941.g4acc112.el8.x86_64 6/6 2020-03-11T22:34:07.026 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-test-2:15.1.0-1941.g4acc112.el8.x86_64 1/6 2020-03-11T22:34:07.026 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : jq-1.5-12.el8.x86_64 2/6 2020-03-11T22:34:07.026 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : oniguruma-6.8.2-1.el8.x86_64 3/6 2020-03-11T22:34:07.027 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : socat-1.7.3.2-6.el8.x86_64 4/6 2020-03-11T22:34:07.027 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libxslt-1.1.32-3.el8.x86_64 5/6 2020-03-11T22:34:07.302 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 6/6 2020-03-11T22:34:07.302 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:34:07.303 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2020-03-11T22:34:07.303 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-test-2:15.1.0-1941.g4acc112.el8.x86_64 jq-1.5-12.el8.x86_64 2020-03-11T22:34:07.303 INFO:teuthology.orchestra.run.smithi145.stdout: oniguruma-6.8.2-1.el8.x86_64 socat-1.7.3.2-6.el8.x86_64 2020-03-11T22:34:07.303 INFO:teuthology.orchestra.run.smithi145.stdout: libxslt-1.1.32-3.el8.x86_64 xmlstarlet-1.6.1-11.el8.x86_64 2020-03-11T22:34:07.303 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:34:07.304 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2020-03-11T22:34:07.567 INFO:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph 2020-03-11T22:34:07.960 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:00:44 ago on Wed 11 Mar 2020 10:33:23 PM UTC. 2020-03-11T22:34:08.371 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2020-03-11T22:34:08.374 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:34:08.375 INFO:teuthology.orchestra.run.smithi145.stdout: Package ArchVersion Repository Size 2020-03-11T22:34:08.375 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:34:08.375 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2020-03-11T22:34:08.375 INFO:teuthology.orchestra.run.smithi145.stdout: ceph x86_642:15.1.0-1941.g4acc112.el8 ceph 6.4 k 2020-03-11T22:34:08.375 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2020-03-11T22:34:08.375 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mds x86_642:15.1.0-1941.g4acc112.el8 ceph 2.0 M 2020-03-11T22:34:08.376 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr x86_642:15.1.0-1941.g4acc112.el8 ceph 1.4 M 2020-03-11T22:34:08.376 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mon x86_642:15.1.0-1941.g4acc112.el8 ceph 4.4 M 2020-03-11T22:34:08.376 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-osd x86_642:15.1.0-1941.g4acc112.el8 ceph 17 M 2020-03-11T22:34:08.376 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-modules-core noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 165 k 2020-03-11T22:34:08.377 INFO:teuthology.orchestra.run.smithi145.stdout: python3-lxml x86_644.2.3-1.el8 CentOS-AppStream 1.5 M 2020-03-11T22:34:08.377 INFO:teuthology.orchestra.run.smithi145.stdout: python3-mako noarch1.0.6-13.el8 CentOS-AppStream 157 k 2020-03-11T22:34:08.377 INFO:teuthology.orchestra.run.smithi145.stdout: python3-markupsafe x86_640.23-19.el8 CentOS-AppStream 39 k 2020-03-11T22:34:08.377 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pytz noarch2017.2-9.el8 CentOS-AppStream 54 k 2020-03-11T22:34:08.377 INFO:teuthology.orchestra.run.smithi145.stdout: python3-webencodings noarch0.5.1-6.el8 CentOS-AppStream 27 k 2020-03-11T22:34:08.377 INFO:teuthology.orchestra.run.smithi145.stdout: python3-werkzeug noarch0.12.2-4.el8 CentOS-AppStream 457 k 2020-03-11T22:34:08.378 INFO:teuthology.orchestra.run.smithi145.stdout: python3-chardet noarch3.0.4-7.el8 CentOS-Base 195 k 2020-03-11T22:34:08.378 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pysocks noarch1.6.8-3.el8 CentOS-Base 34 k 2020-03-11T22:34:08.378 INFO:teuthology.orchestra.run.smithi145.stdout: python3-requests noarch2.20.0-2.1.el8_1 CentOS-Base 123 k 2020-03-11T22:34:08.378 INFO:teuthology.orchestra.run.smithi145.stdout: python3-urllib3 noarch1.24.2-2.el8 CentOS-Base 176 k 2020-03-11T22:34:08.378 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cheroot noarch8.2.1-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 145 k 2020-03-11T22:34:08.378 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cherrypy noarch18.4.0-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 385 k 2020-03-11T22:34:08.379 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco noarch6.2-6.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 11 k 2020-03-11T22:34:08.379 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-functools noarch2.0-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 18 k 2020-03-11T22:34:08.379 INFO:teuthology.orchestra.run.smithi145.stdout: python3-logutils noarch0.3.5-11.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 49 k 2020-03-11T22:34:08.379 INFO:teuthology.orchestra.run.smithi145.stdout: python3-more-itertools noarch7.2.0-3.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 59 k 2020-03-11T22:34:08.379 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pecan noarch1.3.2-9.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 283 k 2020-03-11T22:34:08.380 INFO:teuthology.orchestra.run.smithi145.stdout: python3-portend noarch2.6-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 16 k 2020-03-11T22:34:08.380 INFO:teuthology.orchestra.run.smithi145.stdout: python3-simplegeneric noarch0.8.1-17.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 19 k 2020-03-11T22:34:08.380 INFO:teuthology.orchestra.run.smithi145.stdout: python3-singledispatch noarch3.4.0.3-18.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 24 k 2020-03-11T22:34:08.380 INFO:teuthology.orchestra.run.smithi145.stdout: python3-tempora noarch1.14.1-5.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 30 k 2020-03-11T22:34:08.380 INFO:teuthology.orchestra.run.smithi145.stdout: python3-trustme noarch0.5.2-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 27 k 2020-03-11T22:34:08.380 INFO:teuthology.orchestra.run.smithi145.stdout: python3-zc-lockfile noarch2.0-2.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 23 k 2020-03-11T22:34:08.380 INFO:teuthology.orchestra.run.smithi145.stdout: python3-bcrypt x86_643.1.6-2.el8.1 epel 44 k 2020-03-11T22:34:08.381 INFO:teuthology.orchestra.run.smithi145.stdout: python3-beautifulsoup4 noarch4.6.3-2.el8.1 epel 185 k 2020-03-11T22:34:08.381 INFO:teuthology.orchestra.run.smithi145.stdout: python3-waitress noarch1.2.1-2.el8.1 epel 241 k 2020-03-11T22:34:08.381 INFO:teuthology.orchestra.run.smithi145.stdout: python3-webob noarch1.8.5-1.el8.1 epel 251 k 2020-03-11T22:34:08.381 INFO:teuthology.orchestra.run.smithi145.stdout: python3-webtest noarch2.0.33-1.el8 epel 85 k 2020-03-11T22:34:08.381 INFO:teuthology.orchestra.run.smithi145.stdout:Installing weak dependencies: 2020-03-11T22:34:08.382 INFO:teuthology.orchestra.run.smithi145.stdout: python3-html5lib noarch1:0.999999999-6.el8 CentOS-AppStream 214 k 2020-03-11T22:34:08.382 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cssselect noarch0.9.2-10.el8 epel 40 k 2020-03-11T22:34:08.382 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:34:08.382 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2020-03-11T22:34:08.382 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:34:08.382 INFO:teuthology.orchestra.run.smithi145.stdout:Install 36 Packages 2020-03-11T22:34:08.383 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:34:08.383 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 29 M 2020-03-11T22:34:08.384 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 105 M 2020-03-11T22:34:08.384 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2020-03-11T22:34:08.536 INFO:teuthology.orchestra.run.smithi145.stdout:(1/36): ceph-15.1.0-1941.g4acc112.el8.x86_64.rp 41 kB/s | 6.4 kB 00:00 2020-03-11T22:34:08.821 INFO:teuthology.orchestra.run.smithi145.stdout:(2/36): ceph-mgr-15.1.0-1941.g4acc112.el8.x86_6 3.2 MB/s | 1.4 MB 00:00 2020-03-11T22:34:08.828 INFO:teuthology.orchestra.run.smithi145.stdout:(3/36): ceph-mds-15.1.0-1941.g4acc112.el8.x86_6 4.5 MB/s | 2.0 MB 00:00 2020-03-11T22:34:08.871 INFO:teuthology.orchestra.run.smithi145.stdout:(4/36): ceph-mgr-modules-core-15.1.0-1941.g4acc 3.8 MB/s | 165 kB 00:00 2020-03-11T22:34:08.904 INFO:teuthology.orchestra.run.smithi145.stdout:(5/36): python3-html5lib-0.999999999-6.el8.noar 6.4 MB/s | 214 kB 00:00 2020-03-11T22:34:08.923 INFO:teuthology.orchestra.run.smithi145.stdout:(6/36): python3-lxml-4.2.3-1.el8.x86_64.rpm 78 MB/s | 1.5 MB 00:00 2020-03-11T22:34:08.926 INFO:teuthology.orchestra.run.smithi145.stdout:(7/36): python3-mako-1.0.6-13.el8.noarch.rpm 59 MB/s | 157 kB 00:00 2020-03-11T22:34:08.927 INFO:teuthology.orchestra.run.smithi145.stdout:(8/36): python3-markupsafe-0.23-19.el8.x86_64.r 30 MB/s | 39 kB 00:00 2020-03-11T22:34:08.929 INFO:teuthology.orchestra.run.smithi145.stdout:(9/36): python3-pytz-2017.2-9.el8.noarch.rpm 36 MB/s | 54 kB 00:00 2020-03-11T22:34:08.930 INFO:teuthology.orchestra.run.smithi145.stdout:(10/36): python3-webencodings-0.5.1-6.el8.noarc 24 MB/s | 27 kB 00:00 2020-03-11T22:34:08.936 INFO:teuthology.orchestra.run.smithi145.stdout:(11/36): python3-werkzeug-0.12.2-4.el8.noarch.r 74 MB/s | 457 kB 00:00 2020-03-11T22:34:08.939 INFO:teuthology.orchestra.run.smithi145.stdout:(12/36): python3-chardet-3.0.4-7.el8.noarch.rpm 65 MB/s | 195 kB 00:00 2020-03-11T22:34:08.941 INFO:teuthology.orchestra.run.smithi145.stdout:(13/36): python3-pysocks-1.6.8-3.el8.noarch.rpm 29 MB/s | 34 kB 00:00 2020-03-11T22:34:08.943 INFO:teuthology.orchestra.run.smithi145.stdout:(14/36): python3-requests-2.20.0-2.1.el8_1.noar 54 MB/s | 123 kB 00:00 2020-03-11T22:34:08.946 INFO:teuthology.orchestra.run.smithi145.stdout:(15/36): python3-urllib3-1.24.2-2.el8.noarch.rp 63 MB/s | 176 kB 00:00 2020-03-11T22:34:09.002 INFO:teuthology.orchestra.run.smithi145.stdout:(16/36): python3-cheroot-8.2.1-1.el8.noarch.rpm 2.6 MB/s | 145 kB 00:00 2020-03-11T22:34:09.023 INFO:teuthology.orchestra.run.smithi145.stdout:(17/36): python3-cherrypy-18.4.0-1.el8.noarch.r 17 MB/s | 385 kB 00:00 2020-03-11T22:34:09.033 INFO:teuthology.orchestra.run.smithi145.stdout:(18/36): python3-jaraco-6.2-6.el8.noarch.rpm 1.3 MB/s | 11 kB 00:00 2020-03-11T22:34:09.041 INFO:teuthology.orchestra.run.smithi145.stdout:(19/36): python3-jaraco-functools-2.0-4.el8.noa 2.1 MB/s | 18 kB 00:00 2020-03-11T22:34:09.050 INFO:teuthology.orchestra.run.smithi145.stdout:(20/36): python3-logutils-0.3.5-11.el8.noarch.r 5.7 MB/s | 49 kB 00:00 2020-03-11T22:34:09.059 INFO:teuthology.orchestra.run.smithi145.stdout:(21/36): python3-more-itertools-7.2.0-3.el8.noa 6.4 MB/s | 59 kB 00:00 2020-03-11T22:34:09.074 INFO:teuthology.orchestra.run.smithi145.stdout:(22/36): python3-pecan-1.3.2-9.el8.noarch.rpm 19 MB/s | 283 kB 00:00 2020-03-11T22:34:09.085 INFO:teuthology.orchestra.run.smithi145.stdout:(23/36): python3-portend-2.6-1.el8.noarch.rpm 1.9 MB/s | 16 kB 00:00 2020-03-11T22:34:09.091 INFO:teuthology.orchestra.run.smithi145.stdout:(24/36): python3-simplegeneric-0.8.1-17.el8.noa 2.1 MB/s | 19 kB 00:00 2020-03-11T22:34:09.100 INFO:teuthology.orchestra.run.smithi145.stdout:(25/36): python3-singledispatch-3.4.0.3-18.el8. 2.8 MB/s | 24 kB 00:00 2020-03-11T22:34:09.108 INFO:teuthology.orchestra.run.smithi145.stdout:(26/36): python3-tempora-1.14.1-5.el8.noarch.rp 3.4 MB/s | 30 kB 00:00 2020-03-11T22:34:09.117 INFO:teuthology.orchestra.run.smithi145.stdout:(27/36): python3-trustme-0.5.2-4.el8.noarch.rpm 3.2 MB/s | 27 kB 00:00 2020-03-11T22:34:09.131 INFO:teuthology.orchestra.run.smithi145.stdout:(28/36): ceph-mon-15.1.0-1941.g4acc112.el8.x86_ 7.3 MB/s | 4.4 MB 00:00 2020-03-11T22:34:09.132 INFO:teuthology.orchestra.run.smithi145.stdout:(29/36): python3-zc-lockfile-2.0-2.el8.noarch.r 1.5 MB/s | 23 kB 00:00 2020-03-11T22:34:09.194 INFO:teuthology.orchestra.run.smithi145.stdout:(30/36): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 697 kB/s | 44 kB 00:00 2020-03-11T22:34:09.225 INFO:teuthology.orchestra.run.smithi145.stdout:(31/36): python3-cssselect-0.9.2-10.el8.noarch. 1.3 MB/s | 40 kB 00:00 2020-03-11T22:34:09.226 INFO:teuthology.orchestra.run.smithi145.stdout:(32/36): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.9 MB/s | 185 kB 00:00 2020-03-11T22:34:09.264 INFO:teuthology.orchestra.run.smithi145.stdout:(33/36): python3-webob-1.8.5-1.el8.1.noarch.rpm 6.4 MB/s | 251 kB 00:00 2020-03-11T22:34:09.272 INFO:teuthology.orchestra.run.smithi145.stdout:(34/36): python3-waitress-1.2.1-2.el8.1.noarch. 5.0 MB/s | 241 kB 00:00 2020-03-11T22:34:09.284 INFO:teuthology.orchestra.run.smithi145.stdout:(35/36): python3-webtest-2.0.33-1.el8.noarch.rp 4.3 MB/s | 85 kB 00:00 2020-03-11T22:34:09.847 INFO:teuthology.orchestra.run.smithi145.stdout:(36/36): ceph-osd-15.1.0-1941.g4acc112.el8.x86_ 16 MB/s | 17 MB 00:01 2020-03-11T22:34:09.851 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:34:09.851 INFO:teuthology.orchestra.run.smithi145.stdout:Total 20 MB/s | 29 MB 00:01 2020-03-11T22:34:09.870 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2020-03-11T22:34:09.908 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2020-03-11T22:34:09.908 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2020-03-11T22:34:10.215 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2020-03-11T22:34:10.215 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2020-03-11T22:34:11.437 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2020-03-11T22:34:11.665 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 1/36 2020-03-11T22:34:12.344 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 2/36 2020-03-11T22:34:12.840 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 3/36 2020-03-11T22:34:13.020 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 4/36 2020-03-11T22:34:13.171 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 5/36 2020-03-11T22:34:13.338 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 6/36 2020-03-11T22:34:13.502 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-trustme-0.5.2-4.el8.noarch 7/36 2020-03-11T22:34:13.658 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 8/36 2020-03-11T22:34:13.807 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 9/36 2020-03-11T22:34:13.965 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 10/36 2020-03-11T22:34:14.126 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 11/36 2020-03-11T22:34:14.368 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 12/36 2020-03-11T22:34:14.585 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-cheroot-8.2.1-1.el8.noarch 13/36 2020-03-11T22:34:14.782 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 14/36 2020-03-11T22:34:15.022 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-urllib3-1.24.2-2.el8.noarch 15/36 2020-03-11T22:34:15.234 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 16/36 2020-03-11T22:34:15.454 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 17/36 2020-03-11T22:34:15.646 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 18/36 2020-03-11T22:34:15.904 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 19/36 2020-03-11T22:34:16.131 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 20/36 2020-03-11T22:34:16.464 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 21/36 2020-03-11T22:34:16.719 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-lxml-4.2.3-1.el8.x86_64 22/36 2020-03-11T22:34:16.916 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 23/36 2020-03-11T22:34:17.144 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/36 2020-03-11T22:34:17.281 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 25/36 2020-03-11T22:34:17.500 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-portend-2.6-1.el8.noarch 26/36 2020-03-11T22:34:17.703 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 27/36 2020-03-11T22:34:17.963 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 28/36 2020-03-11T22:34:18.285 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 29/36 2020-03-11T22:34:18.533 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 30/36 2020-03-11T22:34:18.873 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mgr-modules-core-2:15.1.0-1941.g4acc112.el8.n 31/36 2020-03-11T22:34:19.019 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 32/36 2020-03-11T22:34:20.430 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 32/36 2020-03-11T22:34:20.585 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-osd-2:15.1.0-1941.g4acc112.el8.x86_64 33/36 2020-03-11T22:34:20.981 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-osd-2:15.1.0-1941.g4acc112.el8.x86_64 33/36 2020-03-11T22:34:21.110 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mon-2:15.1.0-1941.g4acc112.el8.x86_64 34/36 2020-03-11T22:34:21.302 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mon-2:15.1.0-1941.g4acc112.el8.x86_64 34/36 2020-03-11T22:34:21.477 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mds-2:15.1.0-1941.g4acc112.el8.x86_64 35/36 2020-03-11T22:34:21.495 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mds-2:15.1.0-1941.g4acc112.el8.x86_64 35/36 2020-03-11T22:34:21.608 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-2:15.1.0-1941.g4acc112.el8.x86_64 36/36 2020-03-11T22:34:21.920 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-2:15.1.0-1941.g4acc112.el8.x86_64 36/36 2020-03-11T22:34:21.920 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-2:15.1.0-1941.g4acc112.el8.x86_64 1/36 2020-03-11T22:34:21.921 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mds-2:15.1.0-1941.g4acc112.el8.x86_64 2/36 2020-03-11T22:34:21.921 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 3/36 2020-03-11T22:34:21.921 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mon-2:15.1.0-1941.g4acc112.el8.x86_64 4/36 2020-03-11T22:34:21.921 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-osd-2:15.1.0-1941.g4acc112.el8.x86_64 5/36 2020-03-11T22:34:21.921 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mgr-modules-core-2:15.1.0-1941.g4acc112.el8.n 6/36 2020-03-11T22:34:21.921 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 7/36 2020-03-11T22:34:21.922 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-lxml-4.2.3-1.el8.x86_64 8/36 2020-03-11T22:34:21.922 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 9/36 2020-03-11T22:34:21.922 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 10/36 2020-03-11T22:34:21.922 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 11/36 2020-03-11T22:34:21.922 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 12/36 2020-03-11T22:34:21.922 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 13/36 2020-03-11T22:34:21.922 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 14/36 2020-03-11T22:34:21.923 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 15/36 2020-03-11T22:34:21.923 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 16/36 2020-03-11T22:34:21.923 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-urllib3-1.24.2-2.el8.noarch 17/36 2020-03-11T22:34:21.923 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-cheroot-8.2.1-1.el8.noarch 18/36 2020-03-11T22:34:21.923 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 19/36 2020-03-11T22:34:21.923 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 20/36 2020-03-11T22:34:21.924 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 21/36 2020-03-11T22:34:21.924 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 22/36 2020-03-11T22:34:21.924 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 23/36 2020-03-11T22:34:21.924 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 24/36 2020-03-11T22:34:21.924 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-portend-2.6-1.el8.noarch 25/36 2020-03-11T22:34:21.924 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 26/36 2020-03-11T22:34:21.924 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 27/36 2020-03-11T22:34:21.925 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 28/36 2020-03-11T22:34:21.925 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-trustme-0.5.2-4.el8.noarch 29/36 2020-03-11T22:34:21.925 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 30/36 2020-03-11T22:34:21.925 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 31/36 2020-03-11T22:34:21.925 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 32/36 2020-03-11T22:34:21.925 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 33/36 2020-03-11T22:34:21.926 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 34/36 2020-03-11T22:34:21.926 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 35/36 2020-03-11T22:34:22.637 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 36/36 2020-03-11T22:34:22.637 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:34:22.637 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2020-03-11T22:34:22.637 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:34:22.638 INFO:teuthology.orchestra.run.smithi145.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2020-03-11T22:34:22.638 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cssselect-0.9.2-10.el8.noarch 2020-03-11T22:34:22.638 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mds-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:34:22.638 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:34:22.638 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mon-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:34:22.638 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-osd-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:34:22.638 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-modules-core-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:34:22.639 INFO:teuthology.orchestra.run.smithi145.stdout: python3-lxml-4.2.3-1.el8.x86_64 2020-03-11T22:34:22.639 INFO:teuthology.orchestra.run.smithi145.stdout: python3-mako-1.0.6-13.el8.noarch 2020-03-11T22:34:22.639 INFO:teuthology.orchestra.run.smithi145.stdout: python3-markupsafe-0.23-19.el8.x86_64 2020-03-11T22:34:22.639 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pytz-2017.2-9.el8.noarch 2020-03-11T22:34:22.639 INFO:teuthology.orchestra.run.smithi145.stdout: python3-webencodings-0.5.1-6.el8.noarch 2020-03-11T22:34:22.640 INFO:teuthology.orchestra.run.smithi145.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2020-03-11T22:34:22.640 INFO:teuthology.orchestra.run.smithi145.stdout: python3-chardet-3.0.4-7.el8.noarch 2020-03-11T22:34:22.640 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pysocks-1.6.8-3.el8.noarch 2020-03-11T22:34:22.640 INFO:teuthology.orchestra.run.smithi145.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2020-03-11T22:34:22.640 INFO:teuthology.orchestra.run.smithi145.stdout: python3-urllib3-1.24.2-2.el8.noarch 2020-03-11T22:34:22.640 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cheroot-8.2.1-1.el8.noarch 2020-03-11T22:34:22.640 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2020-03-11T22:34:22.641 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-6.2-6.el8.noarch 2020-03-11T22:34:22.641 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2020-03-11T22:34:22.641 INFO:teuthology.orchestra.run.smithi145.stdout: python3-logutils-0.3.5-11.el8.noarch 2020-03-11T22:34:22.641 INFO:teuthology.orchestra.run.smithi145.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2020-03-11T22:34:22.641 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pecan-1.3.2-9.el8.noarch 2020-03-11T22:34:22.641 INFO:teuthology.orchestra.run.smithi145.stdout: python3-portend-2.6-1.el8.noarch 2020-03-11T22:34:22.642 INFO:teuthology.orchestra.run.smithi145.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2020-03-11T22:34:22.642 INFO:teuthology.orchestra.run.smithi145.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2020-03-11T22:34:22.642 INFO:teuthology.orchestra.run.smithi145.stdout: python3-tempora-1.14.1-5.el8.noarch 2020-03-11T22:34:22.642 INFO:teuthology.orchestra.run.smithi145.stdout: python3-trustme-0.5.2-4.el8.noarch 2020-03-11T22:34:22.642 INFO:teuthology.orchestra.run.smithi145.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2020-03-11T22:34:22.642 INFO:teuthology.orchestra.run.smithi145.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2020-03-11T22:34:22.643 INFO:teuthology.orchestra.run.smithi145.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2020-03-11T22:34:22.643 INFO:teuthology.orchestra.run.smithi145.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2020-03-11T22:34:22.643 INFO:teuthology.orchestra.run.smithi145.stdout: python3-webob-1.8.5-1.el8.1.noarch 2020-03-11T22:34:22.643 INFO:teuthology.orchestra.run.smithi145.stdout: python3-webtest-2.0.33-1.el8.noarch 2020-03-11T22:34:22.643 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:34:22.644 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2020-03-11T22:34:22.956 INFO:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-base 2020-03-11T22:34:23.359 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:00 ago on Wed 11 Mar 2020 10:33:23 PM UTC. 2020-03-11T22:34:23.623 INFO:teuthology.orchestra.run.smithi145.stdout:Package ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:34:23.729 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2020-03-11T22:34:23.730 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2020-03-11T22:34:23.730 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2020-03-11T22:34:23.844 INFO:teuthology.orchestra.run.smithi145:> sudo yum -y install cephadm 2020-03-11T22:34:24.241 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:01 ago on Wed 11 Mar 2020 10:33:23 PM UTC. 2020-03-11T22:34:24.612 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2020-03-11T22:34:24.613 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:34:24.613 INFO:teuthology.orchestra.run.smithi145.stdout: Package Architecture Version Repository Size 2020-03-11T22:34:24.613 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:34:24.614 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2020-03-11T22:34:24.614 INFO:teuthology.orchestra.run.smithi145.stdout: cephadm x86_64 2:15.1.0-1941.g4acc112.el8 ceph 36 k 2020-03-11T22:34:24.614 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:34:24.614 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2020-03-11T22:34:24.614 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:34:24.614 INFO:teuthology.orchestra.run.smithi145.stdout:Install 1 Package 2020-03-11T22:34:24.614 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:34:24.615 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 36 k 2020-03-11T22:34:24.615 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 129 k 2020-03-11T22:34:24.615 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2020-03-11T22:34:24.802 INFO:teuthology.orchestra.run.smithi145.stdout:cephadm-15.1.0-1941.g4acc112.el8.x86_64.rpm 190 kB/s | 36 kB 00:00 2020-03-11T22:34:24.803 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:34:24.803 INFO:teuthology.orchestra.run.smithi145.stdout:Total 188 kB/s | 36 kB 00:00 2020-03-11T22:34:24.805 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2020-03-11T22:34:24.812 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2020-03-11T22:34:24.812 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2020-03-11T22:34:24.814 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2020-03-11T22:34:24.814 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2020-03-11T22:34:24.853 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2020-03-11T22:34:26.130 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: cephadm-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:34:26.400 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : cephadm-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:34:26.610 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : cephadm-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:34:26.610 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:34:26.610 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2020-03-11T22:34:26.610 INFO:teuthology.orchestra.run.smithi145.stdout: cephadm-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:34:26.610 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:34:26.611 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2020-03-11T22:34:26.717 INFO:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-immutable-object-cache 2020-03-11T22:34:27.118 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:04 ago on Wed 11 Mar 2020 10:33:23 PM UTC. 2020-03-11T22:34:27.479 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2020-03-11T22:34:27.479 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:34:27.479 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repo Size 2020-03-11T22:34:27.479 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:34:27.480 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2020-03-11T22:34:27.480 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-immutable-object-cache x86_64 2:15.1.0-1941.g4acc112.el8 ceph 203 k 2020-03-11T22:34:27.480 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:34:27.480 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2020-03-11T22:34:27.480 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:34:27.480 INFO:teuthology.orchestra.run.smithi145.stdout:Install 1 Package 2020-03-11T22:34:27.480 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:34:27.481 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 203 k 2020-03-11T22:34:27.481 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 743 k 2020-03-11T22:34:27.481 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2020-03-11T22:34:27.774 INFO:teuthology.orchestra.run.smithi145.stdout:ceph-immutable-object-cache-15.1.0-1941.g4acc11 692 kB/s | 203 kB 00:00 2020-03-11T22:34:27.774 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:34:27.775 INFO:teuthology.orchestra.run.smithi145.stdout:Total 689 kB/s | 203 kB 00:00 2020-03-11T22:34:27.777 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2020-03-11T22:34:27.789 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2020-03-11T22:34:27.789 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2020-03-11T22:34:27.834 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2020-03-11T22:34:27.835 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2020-03-11T22:34:27.943 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2020-03-11T22:34:28.075 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-immutable-object-cache-2:15.1.0-1941.g4acc112.e 1/1 2020-03-11T22:34:28.732 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-immutable-object-cache-2:15.1.0-1941.g4acc112.e 1/1 2020-03-11T22:34:28.951 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-immutable-object-cache-2:15.1.0-1941.g4acc112.e 1/1 2020-03-11T22:34:28.951 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:34:28.951 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2020-03-11T22:34:28.951 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-immutable-object-cache-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:34:28.952 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:34:28.952 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2020-03-11T22:34:29.207 INFO:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-mgr 2020-03-11T22:34:29.606 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:06 ago on Wed 11 Mar 2020 10:33:23 PM UTC. 2020-03-11T22:34:29.864 INFO:teuthology.orchestra.run.smithi145.stdout:Package ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:34:29.970 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2020-03-11T22:34:29.971 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2020-03-11T22:34:29.971 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2020-03-11T22:34:30.078 INFO:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-mgr-dashboard 2020-03-11T22:34:30.471 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:07 ago on Wed 11 Mar 2020 10:33:23 PM UTC. 2020-03-11T22:34:30.854 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2020-03-11T22:34:30.855 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:34:30.855 INFO:teuthology.orchestra.run.smithi145.stdout: Package ArchVersion Repository Size 2020-03-11T22:34:30.855 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:34:30.855 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2020-03-11T22:34:30.855 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-dashboard noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 3.9 M 2020-03-11T22:34:30.855 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2020-03-11T22:34:30.855 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-grafana-dashboards noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 22 k 2020-03-11T22:34:30.856 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jwt noarch1.6.1-2.el8 CentOS-Base 43 k 2020-03-11T22:34:30.856 INFO:teuthology.orchestra.run.smithi145.stdout: python3-repoze-lru noarch0.7-6.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 34 k 2020-03-11T22:34:30.856 INFO:teuthology.orchestra.run.smithi145.stdout: python3-routes noarch2.4.1-12.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 197 k 2020-03-11T22:34:30.856 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:34:30.856 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2020-03-11T22:34:30.856 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:34:30.856 INFO:teuthology.orchestra.run.smithi145.stdout:Install 5 Packages 2020-03-11T22:34:30.857 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:34:30.857 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 4.2 M 2020-03-11T22:34:30.857 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 19 M 2020-03-11T22:34:30.857 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2020-03-11T22:34:30.887 INFO:teuthology.orchestra.run.smithi145.stdout:(1/5): python3-jwt-1.6.1-2.el8.noarch.rpm 1.4 MB/s | 43 kB 00:00 2020-03-11T22:34:30.951 INFO:teuthology.orchestra.run.smithi145.stdout:(2/5): python3-repoze-lru-0.7-6.el8.noarch.rpm 536 kB/s | 34 kB 00:00 2020-03-11T22:34:30.977 INFO:teuthology.orchestra.run.smithi145.stdout:(3/5): python3-routes-2.4.1-12.el8.noarch.rpm 7.4 MB/s | 197 kB 00:00 2020-03-11T22:34:31.025 INFO:teuthology.orchestra.run.smithi145.stdout:(4/5): ceph-grafana-dashboards-15.1.0-1941.g4ac 133 kB/s | 22 kB 00:00 2020-03-11T22:34:31.279 INFO:teuthology.orchestra.run.smithi145.stdout:(5/5): ceph-mgr-dashboard-15.1.0-1941.g4acc112. 9.2 MB/s | 3.9 MB 00:00 2020-03-11T22:34:31.281 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:34:31.282 INFO:teuthology.orchestra.run.smithi145.stdout:Total 9.8 MB/s | 4.2 MB 00:00 2020-03-11T22:34:31.286 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2020-03-11T22:34:31.295 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2020-03-11T22:34:31.295 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2020-03-11T22:34:31.441 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2020-03-11T22:34:31.442 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2020-03-11T22:34:31.657 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2020-03-11T22:34:31.856 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/5 2020-03-11T22:34:32.084 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/5 2020-03-11T22:34:32.248 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/5 2020-03-11T22:34:33.223 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-grafana-dashboards-2:15.1.0-1941.g4acc112.el8.n 4/5 2020-03-11T22:34:33.474 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mgr-dashboard-2:15.1.0-1941.g4acc112.el8.noarch 5/5 2020-03-11T22:34:33.686 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mgr-dashboard-2:15.1.0-1941.g4acc112.el8.noarch 5/5 2020-03-11T22:34:33.686 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-grafana-dashboards-2:15.1.0-1941.g4acc112.el8.n 1/5 2020-03-11T22:34:33.686 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mgr-dashboard-2:15.1.0-1941.g4acc112.el8.noarch 2/5 2020-03-11T22:34:33.686 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 3/5 2020-03-11T22:34:33.686 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/5 2020-03-11T22:34:34.468 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/5 2020-03-11T22:34:34.469 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:34:34.469 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2020-03-11T22:34:34.469 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-dashboard-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:34:34.469 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-grafana-dashboards-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:34:34.469 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jwt-1.6.1-2.el8.noarch 2020-03-11T22:34:34.469 INFO:teuthology.orchestra.run.smithi145.stdout: python3-repoze-lru-0.7-6.el8.noarch 2020-03-11T22:34:34.470 INFO:teuthology.orchestra.run.smithi145.stdout: python3-routes-2.4.1-12.el8.noarch 2020-03-11T22:34:34.470 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:34:34.470 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2020-03-11T22:34:35.310 INFO:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-mgr-diskprediction-local 2020-03-11T22:34:35.708 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:12 ago on Wed 11 Mar 2020 10:33:23 PM UTC. 2020-03-11T22:34:36.070 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2020-03-11T22:34:36.071 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:34:36.071 INFO:teuthology.orchestra.run.smithi145.stdout: Package ArchVersion Repository Size 2020-03-11T22:34:36.071 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:34:36.072 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2020-03-11T22:34:36.072 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-diskprediction-local noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 8.6 M 2020-03-11T22:34:36.072 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2020-03-11T22:34:36.072 INFO:teuthology.orchestra.run.smithi145.stdout: openblas x86_640.3.3-2.el8 CentOS-AppStream 4.3 M 2020-03-11T22:34:36.072 INFO:teuthology.orchestra.run.smithi145.stdout: openblas-threads x86_640.3.3-2.el8 CentOS-AppStream 4.4 M 2020-03-11T22:34:36.072 INFO:teuthology.orchestra.run.smithi145.stdout: python3-numpy x86_641:1.14.3-9.el8 CentOS-AppStream 3.7 M 2020-03-11T22:34:36.072 INFO:teuthology.orchestra.run.smithi145.stdout: python3-numpy-f2py x86_641:1.14.3-9.el8 CentOS-AppStream 225 k 2020-03-11T22:34:36.073 INFO:teuthology.orchestra.run.smithi145.stdout: python3-scipy x86_641.0.0-20.module_el8.1.0+245+c39af44f CentOS-AppStream 15 M 2020-03-11T22:34:36.073 INFO:teuthology.orchestra.run.smithi145.stdout: libgfortran x86_648.3.1-4.5.el8 CentOS-Base 638 k 2020-03-11T22:34:36.073 INFO:teuthology.orchestra.run.smithi145.stdout: libquadmath x86_648.3.1-4.5.el8 CentOS-Base 169 k 2020-03-11T22:34:36.073 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:34:36.073 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2020-03-11T22:34:36.073 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:34:36.073 INFO:teuthology.orchestra.run.smithi145.stdout:Install 8 Packages 2020-03-11T22:34:36.074 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:34:36.074 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 37 M 2020-03-11T22:34:36.074 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 205 M 2020-03-11T22:34:36.074 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2020-03-11T22:34:36.186 INFO:teuthology.orchestra.run.smithi145.stdout:(1/8): openblas-0.3.3-2.el8.x86_64.rpm 38 MB/s | 4.3 MB 00:00 2020-03-11T22:34:36.218 INFO:teuthology.orchestra.run.smithi145.stdout:(2/8): openblas-threads-0.3.3-2.el8.x86_64.rpm 30 MB/s | 4.4 MB 00:00 2020-03-11T22:34:36.226 INFO:teuthology.orchestra.run.smithi145.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 29 MB/s | 225 kB 00:00 2020-03-11T22:34:36.261 INFO:teuthology.orchestra.run.smithi145.stdout:(4/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 50 MB/s | 3.7 MB 00:00 2020-03-11T22:34:36.274 INFO:teuthology.orchestra.run.smithi145.stdout:(5/8): libgfortran-8.3.1-4.5.el8.x86_64.rpm 47 MB/s | 638 kB 00:00 2020-03-11T22:34:36.280 INFO:teuthology.orchestra.run.smithi145.stdout:(6/8): libquadmath-8.3.1-4.5.el8.x86_64.rpm 29 MB/s | 169 kB 00:00 2020-03-11T22:34:36.447 INFO:teuthology.orchestra.run.smithi145.stdout:(7/8): python3-scipy-1.0.0-20.module_el8.1.0+24 66 MB/s | 15 MB 00:00 2020-03-11T22:34:36.769 INFO:teuthology.orchestra.run.smithi145.stdout:(8/8): ceph-mgr-diskprediction-local-15.1.0-194 12 MB/s | 8.6 MB 00:00 2020-03-11T22:34:36.771 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:34:36.771 INFO:teuthology.orchestra.run.smithi145.stdout:Total 52 MB/s | 37 MB 00:00 2020-03-11T22:34:36.781 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2020-03-11T22:34:36.800 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2020-03-11T22:34:36.800 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2020-03-11T22:34:37.090 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2020-03-11T22:34:37.090 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2020-03-11T22:34:37.633 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2020-03-11T22:34:37.798 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libquadmath-8.3.1-4.5.el8.x86_64 1/8 2020-03-11T22:34:37.934 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: libquadmath-8.3.1-4.5.el8.x86_64 1/8 2020-03-11T22:34:38.081 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libgfortran-8.3.1-4.5.el8.x86_64 2/8 2020-03-11T22:34:38.619 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: libgfortran-8.3.1-4.5.el8.x86_64 2/8 2020-03-11T22:34:38.747 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : openblas-0.3.3-2.el8.x86_64 3/8 2020-03-11T22:34:39.291 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: openblas-0.3.3-2.el8.x86_64 3/8 2020-03-11T22:34:39.432 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : openblas-threads-0.3.3-2.el8.x86_64 4/8 2020-03-11T22:34:40.131 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: openblas-threads-0.3.3-2.el8.x86_64 4/8 2020-03-11T22:34:40.418 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2020-03-11T22:34:41.148 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2020-03-11T22:34:43.109 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2020-03-11T22:34:44.365 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x 7/8 2020-03-11T22:34:44.539 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mgr-diskprediction-local-2:15.1.0-1941.g4acc112 8/8 2020-03-11T22:34:44.788 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:15.1.0-1941.g4acc112 8/8 2020-03-11T22:34:44.788 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mgr-diskprediction-local-2:15.1.0-1941.g4acc112 1/8 2020-03-11T22:34:44.788 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : openblas-0.3.3-2.el8.x86_64 2/8 2020-03-11T22:34:44.788 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : openblas-threads-0.3.3-2.el8.x86_64 3/8 2020-03-11T22:34:44.789 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2020-03-11T22:34:44.789 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2020-03-11T22:34:44.789 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x 6/8 2020-03-11T22:34:44.789 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libgfortran-8.3.1-4.5.el8.x86_64 7/8 2020-03-11T22:34:45.074 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libquadmath-8.3.1-4.5.el8.x86_64 8/8 2020-03-11T22:34:45.074 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:34:45.074 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2020-03-11T22:34:45.074 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-diskprediction-local-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:34:45.074 INFO:teuthology.orchestra.run.smithi145.stdout: openblas-0.3.3-2.el8.x86_64 2020-03-11T22:34:45.075 INFO:teuthology.orchestra.run.smithi145.stdout: openblas-threads-0.3.3-2.el8.x86_64 2020-03-11T22:34:45.075 INFO:teuthology.orchestra.run.smithi145.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2020-03-11T22:34:45.075 INFO:teuthology.orchestra.run.smithi145.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2020-03-11T22:34:45.075 INFO:teuthology.orchestra.run.smithi145.stdout: python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x86_64 2020-03-11T22:34:45.075 INFO:teuthology.orchestra.run.smithi145.stdout: libgfortran-8.3.1-4.5.el8.x86_64 2020-03-11T22:34:45.076 INFO:teuthology.orchestra.run.smithi145.stdout: libquadmath-8.3.1-4.5.el8.x86_64 2020-03-11T22:34:45.076 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:34:45.076 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2020-03-11T22:34:45.186 INFO:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-mgr-rook 2020-03-11T22:34:45.582 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:22 ago on Wed 11 Mar 2020 10:33:23 PM UTC. 2020-03-11T22:34:45.941 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2020-03-11T22:34:45.943 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:34:45.943 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2020-03-11T22:34:45.943 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:34:45.944 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2020-03-11T22:34:45.944 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-rook noarch 2:15.1.0-1941.g4acc112.el8 2020-03-11T22:34:45.944 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-noarch 24 k 2020-03-11T22:34:45.944 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2020-03-11T22:34:45.944 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2020-03-11T22:34:45.945 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2020-03-11T22:34:45.945 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2020-03-11T22:34:45.945 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2020-03-11T22:34:45.945 INFO:teuthology.orchestra.run.smithi145.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2020-03-11T22:34:45.945 INFO:teuthology.orchestra.run.smithi145.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2020-03-11T22:34:45.945 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2020-03-11T22:34:45.946 INFO:teuthology.orchestra.run.smithi145.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2020-03-11T22:34:45.946 INFO:teuthology.orchestra.run.smithi145.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2020-03-11T22:34:45.946 INFO:teuthology.orchestra.run.smithi145.stdout: python3-kubernetes noarch 10.0.1-2.el8 epel 1.1 M 2020-03-11T22:34:45.946 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rsa noarch 3.4.2-12.el8 epel 73 k 2020-03-11T22:34:45.946 INFO:teuthology.orchestra.run.smithi145.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2020-03-11T22:34:45.946 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:34:45.947 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2020-03-11T22:34:45.947 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:34:45.947 INFO:teuthology.orchestra.run.smithi145.stdout:Install 13 Packages 2020-03-11T22:34:45.947 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:34:45.948 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 1.9 M 2020-03-11T22:34:45.948 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 23 M 2020-03-11T22:34:45.948 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2020-03-11T22:34:45.976 INFO:teuthology.orchestra.run.smithi145.stdout:(1/13): python3-jsonpointer-1.10-11.el8.noarch. 684 kB/s | 20 kB 00:00 2020-03-11T22:34:45.979 INFO:teuthology.orchestra.run.smithi145.stdout:(2/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 43 MB/s | 126 kB 00:00 2020-03-11T22:34:45.980 INFO:teuthology.orchestra.run.smithi145.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 789 kB/s | 27 kB 00:00 2020-03-11T22:34:45.982 INFO:teuthology.orchestra.run.smithi145.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 41 MB/s | 110 kB 00:00 2020-03-11T22:34:45.988 INFO:teuthology.orchestra.run.smithi145.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 7.0 MB/s | 43 kB 00:00 2020-03-11T22:34:45.988 INFO:teuthology.orchestra.run.smithi145.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 19 MB/s | 155 kB 00:00 2020-03-11T22:34:46.046 INFO:teuthology.orchestra.run.smithi145.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 285 kB/s | 16 kB 00:00 2020-03-11T22:34:46.046 INFO:teuthology.orchestra.run.smithi145.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 564 kB/s | 33 kB 00:00 2020-03-11T22:34:46.084 INFO:teuthology.orchestra.run.smithi145.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 2.1 MB/s | 81 kB 00:00 2020-03-11T22:34:46.103 INFO:teuthology.orchestra.run.smithi145.stdout:(10/13): python3-rsa-3.4.2-12.el8.noarch.rpm 3.7 MB/s | 73 kB 00:00 2020-03-11T22:34:46.122 INFO:teuthology.orchestra.run.smithi145.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 3.1 MB/s | 61 kB 00:00 2020-03-11T22:34:46.133 INFO:teuthology.orchestra.run.smithi145.stdout:(12/13): ceph-mgr-rook-15.1.0-1941.g4acc112.el8 131 kB/s | 24 kB 00:00 2020-03-11T22:34:46.179 INFO:teuthology.orchestra.run.smithi145.stdout:(13/13): python3-kubernetes-10.0.1-2.el8.noarch 8.4 MB/s | 1.1 MB 00:00 2020-03-11T22:34:46.181 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:34:46.181 INFO:teuthology.orchestra.run.smithi145.stdout:Total 7.9 MB/s | 1.9 MB 00:00 2020-03-11T22:34:46.188 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2020-03-11T22:34:46.202 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2020-03-11T22:34:46.202 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2020-03-11T22:34:46.287 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2020-03-11T22:34:46.288 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2020-03-11T22:34:46.741 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2020-03-11T22:34:47.016 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2020-03-11T22:34:47.244 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2020-03-11T22:34:47.408 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-rsa-3.4.2-12.el8.noarch 3/13 2020-03-11T22:34:47.586 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2020-03-11T22:34:47.777 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2020-03-11T22:34:47.966 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2020-03-11T22:34:48.215 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2020-03-11T22:34:48.467 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2020-03-11T22:34:49.484 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2020-03-11T22:34:49.686 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-kubernetes-10.0.1-2.el8.noarch 10/13 2020-03-11T22:34:49.903 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2020-03-11T22:34:50.087 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2020-03-11T22:34:50.254 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mgr-rook-2:15.1.0-1941.g4acc112.el8.noarch 13/13 2020-03-11T22:34:50.469 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mgr-rook-2:15.1.0-1941.g4acc112.el8.noarch 13/13 2020-03-11T22:34:50.469 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mgr-rook-2:15.1.0-1941.g4acc112.el8.noarch 1/13 2020-03-11T22:34:50.469 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2020-03-11T22:34:50.470 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2020-03-11T22:34:50.470 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2020-03-11T22:34:50.470 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2020-03-11T22:34:50.470 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2020-03-11T22:34:50.470 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2020-03-11T22:34:50.470 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2020-03-11T22:34:50.470 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2020-03-11T22:34:50.471 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2020-03-11T22:34:50.471 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-kubernetes-10.0.1-2.el8.noarch 11/13 2020-03-11T22:34:50.471 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-rsa-3.4.2-12.el8.noarch 12/13 2020-03-11T22:34:50.781 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2020-03-11T22:34:50.781 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:34:50.781 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2020-03-11T22:34:50.781 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-rook-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:34:50.781 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jsonpatch-1.21-2.el8.noarch 2020-03-11T22:34:50.782 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jsonpointer-1.10-11.el8.noarch 2020-03-11T22:34:50.782 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2020-03-11T22:34:50.782 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2020-03-11T22:34:50.782 INFO:teuthology.orchestra.run.smithi145.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2020-03-11T22:34:50.782 INFO:teuthology.orchestra.run.smithi145.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2020-03-11T22:34:50.782 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cachetools-3.1.1-4.el8.noarch 2020-03-11T22:34:50.782 INFO:teuthology.orchestra.run.smithi145.stdout: python3-certifi-2018.10.15-7.el8.noarch 2020-03-11T22:34:50.783 INFO:teuthology.orchestra.run.smithi145.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2020-03-11T22:34:50.783 INFO:teuthology.orchestra.run.smithi145.stdout: python3-kubernetes-10.0.1-2.el8.noarch 2020-03-11T22:34:50.783 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rsa-3.4.2-12.el8.noarch 2020-03-11T22:34:50.783 INFO:teuthology.orchestra.run.smithi145.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2020-03-11T22:34:50.783 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:34:50.783 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2020-03-11T22:34:50.908 INFO:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-mgr-cephadm 2020-03-11T22:34:51.306 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:28 ago on Wed 11 Mar 2020 10:33:23 PM UTC. 2020-03-11T22:34:52.211 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2020-03-11T22:34:52.212 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:34:52.212 INFO:teuthology.orchestra.run.smithi145.stdout: Package ArchVersion Repository Size 2020-03-11T22:34:52.212 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:34:52.212 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2020-03-11T22:34:52.212 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-cephadm noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 36 k 2020-03-11T22:34:52.213 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2020-03-11T22:34:52.213 INFO:teuthology.orchestra.run.smithi145.stdout: python3-execnet noarch1.7.1-1.el8.2 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 356 k 2020-03-11T22:34:52.213 INFO:teuthology.orchestra.run.smithi145.stdout: python3-remoto noarch1.1.4-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 48 k 2020-03-11T22:34:52.213 INFO:teuthology.orchestra.run.smithi145.stdout: python3-apipkg noarch1.5-6.el8 epel 21 k 2020-03-11T22:34:52.213 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:34:52.213 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2020-03-11T22:34:52.213 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:34:52.214 INFO:teuthology.orchestra.run.smithi145.stdout:Install 4 Packages 2020-03-11T22:34:52.214 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:34:52.214 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 461 k 2020-03-11T22:34:52.214 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 1.4 M 2020-03-11T22:34:52.215 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2020-03-11T22:34:52.272 INFO:teuthology.orchestra.run.smithi145.stdout:(1/4): python3-remoto-1.1.4-4.el8.noarch.rpm 841 kB/s | 48 kB 00:00 2020-03-11T22:34:52.279 INFO:teuthology.orchestra.run.smithi145.stdout:(2/4): python3-execnet-1.7.1-1.el8.2.noarch.rpm 5.4 MB/s | 356 kB 00:00 2020-03-11T22:34:52.319 INFO:teuthology.orchestra.run.smithi145.stdout:(3/4): python3-apipkg-1.5-6.el8.noarch.rpm 448 kB/s | 21 kB 00:00 2020-03-11T22:34:52.402 INFO:teuthology.orchestra.run.smithi145.stdout:(4/4): ceph-mgr-cephadm-15.1.0-1941.g4acc112.el 194 kB/s | 36 kB 00:00 2020-03-11T22:34:52.403 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:34:52.405 INFO:teuthology.orchestra.run.smithi145.stdout:Total 2.4 MB/s | 461 kB 00:00 2020-03-11T22:34:52.407 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2020-03-11T22:34:52.414 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2020-03-11T22:34:52.414 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2020-03-11T22:34:52.443 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2020-03-11T22:34:52.444 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2020-03-11T22:34:52.599 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2020-03-11T22:34:52.845 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/4 2020-03-11T22:34:53.087 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-execnet-1.7.1-1.el8.2.noarch 2/4 2020-03-11T22:34:53.251 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 3/4 2020-03-11T22:34:53.412 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mgr-cephadm-2:15.1.0-1941.g4acc112.el8.noarch 4/4 2020-03-11T22:34:53.623 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mgr-cephadm-2:15.1.0-1941.g4acc112.el8.noarch 4/4 2020-03-11T22:34:53.623 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mgr-cephadm-2:15.1.0-1941.g4acc112.el8.noarch 1/4 2020-03-11T22:34:53.623 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-execnet-1.7.1-1.el8.2.noarch 2/4 2020-03-11T22:34:53.623 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 3/4 2020-03-11T22:34:53.855 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/4 2020-03-11T22:34:53.855 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:34:53.855 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2020-03-11T22:34:53.855 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-cephadm-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:34:53.855 INFO:teuthology.orchestra.run.smithi145.stdout: python3-execnet-1.7.1-1.el8.2.noarch 2020-03-11T22:34:53.855 INFO:teuthology.orchestra.run.smithi145.stdout: python3-remoto-1.1.4-4.el8.noarch 2020-03-11T22:34:53.856 INFO:teuthology.orchestra.run.smithi145.stdout: python3-apipkg-1.5-6.el8.noarch 2020-03-11T22:34:53.856 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:34:53.856 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2020-03-11T22:34:53.966 INFO:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-fuse 2020-03-11T22:34:54.371 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:31 ago on Wed 11 Mar 2020 10:33:23 PM UTC. 2020-03-11T22:34:54.749 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2020-03-11T22:34:54.749 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:34:54.750 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2020-03-11T22:34:54.750 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:34:54.750 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2020-03-11T22:34:54.751 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-fuse x86_64 2:15.1.0-1941.g4acc112.el8 ceph 572 k 2020-03-11T22:34:54.751 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2020-03-11T22:34:54.751 INFO:teuthology.orchestra.run.smithi145.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2020-03-11T22:34:54.751 INFO:teuthology.orchestra.run.smithi145.stdout: fuse-common x86_64 3.2.1-12.el8 CentOS-Base 21 k 2020-03-11T22:34:54.752 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:34:54.752 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2020-03-11T22:34:54.752 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:34:54.752 INFO:teuthology.orchestra.run.smithi145.stdout:Install 3 Packages 2020-03-11T22:34:54.752 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:34:54.753 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 676 k 2020-03-11T22:34:54.754 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 2.1 M 2020-03-11T22:34:54.754 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2020-03-11T22:34:54.783 INFO:teuthology.orchestra.run.smithi145.stdout:(1/3): fuse-2.9.7-12.el8.x86_64.rpm 2.5 MB/s | 82 kB 00:00 2020-03-11T22:34:54.789 INFO:teuthology.orchestra.run.smithi145.stdout:(2/3): fuse-common-3.2.1-12.el8.x86_64.rpm 551 kB/s | 21 kB 00:00 2020-03-11T22:34:55.040 INFO:teuthology.orchestra.run.smithi145.stdout:(3/3): ceph-fuse-15.1.0-1941.g4acc112.el8.x86_6 1.9 MB/s | 572 kB 00:00 2020-03-11T22:34:55.041 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:34:55.042 INFO:teuthology.orchestra.run.smithi145.stdout:Total 2.3 MB/s | 676 kB 00:00 2020-03-11T22:34:55.046 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2020-03-11T22:34:55.066 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2020-03-11T22:34:55.066 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2020-03-11T22:34:55.134 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2020-03-11T22:34:55.134 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2020-03-11T22:34:55.298 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2020-03-11T22:34:55.299 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 1/3 2020-03-11T22:34:55.299 INFO:teuthology.orchestra.run.smithi145.stdout:warning: /etc/fuse.conf created as /etc/fuse.conf.rpmnew 2020-03-11T22:34:55.299 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:34:55.694 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : fuse-2.9.7-12.el8.x86_64 2/3 2020-03-11T22:34:55.860 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 3/3 2020-03-11T22:34:56.162 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 3/3 2020-03-11T22:34:56.162 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 1/3 2020-03-11T22:34:56.162 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/3 2020-03-11T22:34:56.430 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/3 2020-03-11T22:34:56.430 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:34:56.430 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2020-03-11T22:34:56.430 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2020-03-11T22:34:56.430 INFO:teuthology.orchestra.run.smithi145.stdout: fuse-common-3.2.1-12.el8.x86_64 2020-03-11T22:34:56.431 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:34:56.431 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2020-03-11T22:34:56.705 INFO:teuthology.orchestra.run.smithi145:> sudo yum -y install librados-devel 2020-03-11T22:34:57.110 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:34 ago on Wed 11 Mar 2020 10:33:23 PM UTC. 2020-03-11T22:34:57.863 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2020-03-11T22:34:57.864 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:34:57.864 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repo Size 2020-03-11T22:34:57.864 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:34:57.865 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2020-03-11T22:34:57.865 INFO:teuthology.orchestra.run.smithi145.stdout: librados-devel x86_64 2:15.1.0-1941.g4acc112.el8 ceph 141 k 2020-03-11T22:34:57.865 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:34:57.865 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2020-03-11T22:34:57.866 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:34:57.866 INFO:teuthology.orchestra.run.smithi145.stdout:Install 1 Package 2020-03-11T22:34:57.866 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:34:57.866 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 141 k 2020-03-11T22:34:57.867 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 487 k 2020-03-11T22:34:57.867 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2020-03-11T22:34:58.295 INFO:teuthology.orchestra.run.smithi145.stdout:librados-devel-15.1.0-1941.g4acc112.el8.x86_64. 326 kB/s | 141 kB 00:00 2020-03-11T22:34:58.296 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:34:58.296 INFO:teuthology.orchestra.run.smithi145.stdout:Total 325 kB/s | 141 kB 00:00 2020-03-11T22:34:58.299 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2020-03-11T22:34:58.306 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2020-03-11T22:34:58.306 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2020-03-11T22:34:58.359 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2020-03-11T22:34:58.360 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2020-03-11T22:34:58.473 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2020-03-11T22:34:58.659 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : librados-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:34:58.862 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: librados-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:34:59.129 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librados-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:34:59.129 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:34:59.129 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2020-03-11T22:34:59.130 INFO:teuthology.orchestra.run.smithi145.stdout: librados-devel-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:34:59.130 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:34:59.130 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2020-03-11T22:34:59.396 INFO:teuthology.orchestra.run.smithi145:> sudo yum -y install libcephfs2 2020-03-11T22:34:59.799 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:36 ago on Wed 11 Mar 2020 10:33:23 PM UTC. 2020-03-11T22:35:00.075 INFO:teuthology.orchestra.run.smithi145.stdout:Package libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:35:00.183 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2020-03-11T22:35:00.184 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2020-03-11T22:35:00.184 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2020-03-11T22:35:00.295 INFO:teuthology.orchestra.run.smithi145:> sudo yum -y install libcephfs-devel 2020-03-11T22:35:00.695 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:37 ago on Wed 11 Mar 2020 10:33:23 PM UTC. 2020-03-11T22:35:01.074 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2020-03-11T22:35:01.075 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:35:01.075 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repo Size 2020-03-11T22:35:01.075 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:35:01.076 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2020-03-11T22:35:01.076 INFO:teuthology.orchestra.run.smithi145.stdout: libcephfs-devel x86_64 2:15.1.0-1941.g4acc112.el8 ceph 21 k 2020-03-11T22:35:01.076 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:35:01.076 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2020-03-11T22:35:01.076 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:35:01.076 INFO:teuthology.orchestra.run.smithi145.stdout:Install 1 Package 2020-03-11T22:35:01.076 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:35:01.077 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 21 k 2020-03-11T22:35:01.077 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 76 k 2020-03-11T22:35:01.077 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2020-03-11T22:35:01.241 INFO:teuthology.orchestra.run.smithi145.stdout:libcephfs-devel-15.1.0-1941.g4acc112.el8.x86_64 129 kB/s | 21 kB 00:00 2020-03-11T22:35:01.241 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:35:01.241 INFO:teuthology.orchestra.run.smithi145.stdout:Total 129 kB/s | 21 kB 00:00 2020-03-11T22:35:01.243 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2020-03-11T22:35:01.247 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2020-03-11T22:35:01.247 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2020-03-11T22:35:01.249 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2020-03-11T22:35:01.250 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2020-03-11T22:35:01.303 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2020-03-11T22:35:01.450 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libcephfs-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:35:01.640 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: libcephfs-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:35:01.853 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libcephfs-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:35:01.853 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:35:01.853 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2020-03-11T22:35:01.854 INFO:teuthology.orchestra.run.smithi145.stdout: libcephfs-devel-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:35:01.854 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:35:01.854 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2020-03-11T22:35:01.998 INFO:teuthology.orchestra.run.smithi145:> sudo yum -y install librados2 2020-03-11T22:35:02.398 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:39 ago on Wed 11 Mar 2020 10:33:23 PM UTC. 2020-03-11T22:35:02.658 INFO:teuthology.orchestra.run.smithi145.stdout:Package librados2-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:35:02.765 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2020-03-11T22:35:02.766 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2020-03-11T22:35:02.766 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2020-03-11T22:35:02.875 INFO:teuthology.orchestra.run.smithi145:> sudo yum -y install librbd1 2020-03-11T22:35:03.276 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:40 ago on Wed 11 Mar 2020 10:33:23 PM UTC. 2020-03-11T22:35:03.541 INFO:teuthology.orchestra.run.smithi145.stdout:Package librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:35:03.647 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2020-03-11T22:35:03.648 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2020-03-11T22:35:03.648 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2020-03-11T22:35:03.746 INFO:teuthology.orchestra.run.smithi145:> sudo yum -y install python3-rados 2020-03-11T22:35:04.145 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:41 ago on Wed 11 Mar 2020 10:33:23 PM UTC. 2020-03-11T22:35:04.400 INFO:teuthology.orchestra.run.smithi145.stdout:Package python3-rados-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:35:04.506 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2020-03-11T22:35:04.507 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2020-03-11T22:35:04.507 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2020-03-11T22:35:04.619 INFO:teuthology.orchestra.run.smithi145:> sudo yum -y install python3-rgw 2020-03-11T22:35:05.019 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:42 ago on Wed 11 Mar 2020 10:33:23 PM UTC. 2020-03-11T22:35:05.283 INFO:teuthology.orchestra.run.smithi145.stdout:Package python3-rgw-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:35:05.391 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2020-03-11T22:35:05.392 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2020-03-11T22:35:05.393 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2020-03-11T22:35:05.499 INFO:teuthology.orchestra.run.smithi145:> sudo yum -y install python3-cephfs 2020-03-11T22:35:05.898 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:42 ago on Wed 11 Mar 2020 10:33:23 PM UTC. 2020-03-11T22:35:06.166 INFO:teuthology.orchestra.run.smithi145.stdout:Package python3-cephfs-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:35:06.272 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2020-03-11T22:35:06.273 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2020-03-11T22:35:06.273 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2020-03-11T22:35:06.383 INFO:teuthology.orchestra.run.smithi145:> sudo yum -y install python3-rbd 2020-03-11T22:35:06.804 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:43 ago on Wed 11 Mar 2020 10:33:23 PM UTC. 2020-03-11T22:35:07.049 INFO:teuthology.orchestra.run.smithi145.stdout:Package python3-rbd-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:35:07.153 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2020-03-11T22:35:07.155 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2020-03-11T22:35:07.155 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2020-03-11T22:35:07.262 INFO:teuthology.orchestra.run.smithi145:> sudo yum -y install rbd-fuse 2020-03-11T22:35:07.665 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:44 ago on Wed 11 Mar 2020 10:33:23 PM UTC. 2020-03-11T22:35:08.032 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2020-03-11T22:35:08.033 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:35:08.033 INFO:teuthology.orchestra.run.smithi145.stdout: Package Architecture Version Repository Size 2020-03-11T22:35:08.033 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:35:08.033 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2020-03-11T22:35:08.033 INFO:teuthology.orchestra.run.smithi145.stdout: rbd-fuse x86_64 2:15.1.0-1941.g4acc112.el8 ceph 72 k 2020-03-11T22:35:08.034 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:35:08.034 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2020-03-11T22:35:08.034 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:35:08.034 INFO:teuthology.orchestra.run.smithi145.stdout:Install 1 Package 2020-03-11T22:35:08.034 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:35:08.035 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 72 k 2020-03-11T22:35:08.035 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 230 k 2020-03-11T22:35:08.035 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2020-03-11T22:35:08.238 INFO:teuthology.orchestra.run.smithi145.stdout:rbd-fuse-15.1.0-1941.g4acc112.el8.x86_64.rpm 354 kB/s | 72 kB 00:00 2020-03-11T22:35:08.239 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:35:08.239 INFO:teuthology.orchestra.run.smithi145.stdout:Total 351 kB/s | 72 kB 00:00 2020-03-11T22:35:08.241 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2020-03-11T22:35:08.252 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2020-03-11T22:35:08.253 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2020-03-11T22:35:08.301 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2020-03-11T22:35:08.302 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2020-03-11T22:35:08.790 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2020-03-11T22:35:08.949 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : rbd-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:35:09.137 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: rbd-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:35:09.402 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : rbd-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:35:09.402 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:35:09.402 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2020-03-11T22:35:09.403 INFO:teuthology.orchestra.run.smithi145.stdout: rbd-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:35:09.403 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:35:09.403 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2020-03-11T22:35:09.664 INFO:teuthology.orchestra.run.smithi145:> sudo yum -y install rbd-mirror 2020-03-11T22:35:10.067 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:47 ago on Wed 11 Mar 2020 10:33:23 PM UTC. 2020-03-11T22:35:10.482 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2020-03-11T22:35:10.482 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:35:10.482 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repo Size 2020-03-11T22:35:10.482 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:35:10.483 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2020-03-11T22:35:10.483 INFO:teuthology.orchestra.run.smithi145.stdout: rbd-mirror x86_64 2:15.1.0-1941.g4acc112.el8 ceph 2.7 M 2020-03-11T22:35:10.483 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:35:10.483 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2020-03-11T22:35:10.483 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:35:10.483 INFO:teuthology.orchestra.run.smithi145.stdout:Install 1 Package 2020-03-11T22:35:10.483 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:35:10.484 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 2.7 M 2020-03-11T22:35:10.484 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 12 M 2020-03-11T22:35:10.484 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2020-03-11T22:35:10.928 INFO:teuthology.orchestra.run.smithi145.stdout:rbd-mirror-15.1.0-1941.g4acc112.el8.x86_64.rpm 6.0 MB/s | 2.7 MB 00:00 2020-03-11T22:35:10.928 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:35:10.929 INFO:teuthology.orchestra.run.smithi145.stdout:Total 6.0 MB/s | 2.7 MB 00:00 2020-03-11T22:35:10.931 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2020-03-11T22:35:10.943 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2020-03-11T22:35:10.943 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2020-03-11T22:35:11.001 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2020-03-11T22:35:11.001 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2020-03-11T22:35:11.367 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2020-03-11T22:35:11.517 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : rbd-mirror-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:35:11.838 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: rbd-mirror-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:35:12.093 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : rbd-mirror-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:35:12.093 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:35:12.094 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2020-03-11T22:35:12.094 INFO:teuthology.orchestra.run.smithi145.stdout: rbd-mirror-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:35:12.094 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:35:12.094 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2020-03-11T22:35:12.375 INFO:teuthology.orchestra.run.smithi145:> sudo yum -y install rbd-nbd 2020-03-11T22:35:12.778 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:49 ago on Wed 11 Mar 2020 10:33:23 PM UTC. 2020-03-11T22:35:13.146 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2020-03-11T22:35:13.147 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:35:13.148 INFO:teuthology.orchestra.run.smithi145.stdout: Package Architecture Version Repository Size 2020-03-11T22:35:13.148 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:35:13.148 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2020-03-11T22:35:13.148 INFO:teuthology.orchestra.run.smithi145.stdout: rbd-nbd x86_64 2:15.1.0-1941.g4acc112.el8 ceph 141 k 2020-03-11T22:35:13.148 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:35:13.148 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2020-03-11T22:35:13.148 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-03-11T22:35:13.149 INFO:teuthology.orchestra.run.smithi145.stdout:Install 1 Package 2020-03-11T22:35:13.149 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:35:13.149 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 141 k 2020-03-11T22:35:13.149 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 526 k 2020-03-11T22:35:13.149 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2020-03-11T22:35:13.407 INFO:teuthology.orchestra.run.smithi145.stdout:rbd-nbd-15.1.0-1941.g4acc112.el8.x86_64.rpm 544 kB/s | 141 kB 00:00 2020-03-11T22:35:13.408 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:35:13.408 INFO:teuthology.orchestra.run.smithi145.stdout:Total 541 kB/s | 141 kB 00:00 2020-03-11T22:35:13.410 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2020-03-11T22:35:13.421 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2020-03-11T22:35:13.421 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2020-03-11T22:35:13.469 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2020-03-11T22:35:13.469 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2020-03-11T22:35:13.582 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2020-03-11T22:35:14.306 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : rbd-nbd-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:35:14.800 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: rbd-nbd-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:35:15.526 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : rbd-nbd-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:35:15.527 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:35:15.527 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2020-03-11T22:35:15.527 INFO:teuthology.orchestra.run.smithi145.stdout: rbd-nbd-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:35:15.527 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-03-11T22:35:15.527 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2020-03-11T22:35:15.681 DEBUG:teuthology.parallel:result is None 2020-03-11T22:35:15.681 INFO:teuthology.orchestra.run.smithi145:> sudo lsb_release -is 2020-03-11T22:35:15.718 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS 2020-03-11T22:35:15.719 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-11T22:35:15.720 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-11T22:35:15.930 INFO:teuthology.orchestra.run.smithi145:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2020-03-11T22:35:15.949 INFO:teuthology.orchestra.run.smithi145.stdout:15.1.0-1941.g4acc112.el8 2020-03-11T22:35:15.949 INFO:teuthology.packaging:The installed version of ceph is 15.1.0-1941.g4acc112.el8 2020-03-11T22:35:15.949 INFO:teuthology.task.install:The correct ceph version 15.1.0-1941.g4acc112 is installed. 2020-03-11T22:35:15.950 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2020-03-11T22:35:15.950 INFO:teuthology.orchestra.run.smithi145:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2020-03-11T22:35:16.019 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2020-03-11T22:35:16.019 INFO:teuthology.orchestra.run.smithi145:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2020-03-11T22:35:16.087 INFO:teuthology.orchestra.run.smithi145:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-03-11T22:35:16.153 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2020-03-11T22:35:16.154 INFO:teuthology.orchestra.run.smithi145:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2020-03-11T22:35:16.239 INFO:teuthology.orchestra.run.smithi145:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-03-11T22:35:16.306 INFO:teuthology.run_tasks:Running task workunit... 2020-03-11T22:35:16.320 INFO:tasks.workunit:Pulling workunits from ref 4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-11T22:35:16.321 INFO:tasks.workunit:Making a separate scratch dir for every client... 2020-03-11T22:35:16.321 INFO:tasks.workunit:timeout=3h 2020-03-11T22:35:16.321 INFO:tasks.workunit:cleanup=True 2020-03-11T22:35:16.321 INFO:teuthology.orchestra.run.smithi145:> stat -- /home/ubuntu/cephtest/mnt.0 2020-03-11T22:35:16.359 INFO:teuthology.orchestra.run.smithi145.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2020-03-11T22:35:16.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-03-11T22:35:16.360 INFO:teuthology.orchestra.run.smithi145:> mkdir -- /home/ubuntu/cephtest/mnt.0 2020-03-11T22:35:16.413 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2020-03-11T22:35:16.414 INFO:teuthology.orchestra.run.smithi145:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2020-03-11T22:35:16.467 INFO:teuthology.orchestra.run.smithi145:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-11T22:35:16.583 INFO:tasks.workunit.client.0.smithi145.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2020-03-11T22:36:55.761 INFO:tasks.workunit.client.0.smithi145.stderr:Note: checking out '4acc1121e47039a3cdf2978ae458c10d153aa141'. 2020-03-11T22:36:55.761 INFO:tasks.workunit.client.0.smithi145.stderr: 2020-03-11T22:36:55.761 INFO:tasks.workunit.client.0.smithi145.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2020-03-11T22:36:55.762 INFO:tasks.workunit.client.0.smithi145.stderr:changes and commit them, and you can discard any commits you make in this 2020-03-11T22:36:55.762 INFO:tasks.workunit.client.0.smithi145.stderr:state without impacting any branches by performing another checkout. 2020-03-11T22:36:55.762 INFO:tasks.workunit.client.0.smithi145.stderr: 2020-03-11T22:36:55.762 INFO:tasks.workunit.client.0.smithi145.stderr:If you want to create a new branch to retain commits you create, you may 2020-03-11T22:36:55.763 INFO:tasks.workunit.client.0.smithi145.stderr:do so (now or later) by using -b with the checkout command again. Example: 2020-03-11T22:36:55.763 INFO:tasks.workunit.client.0.smithi145.stderr: 2020-03-11T22:36:55.763 INFO:tasks.workunit.client.0.smithi145.stderr: git checkout -b 2020-03-11T22:36:55.763 INFO:tasks.workunit.client.0.smithi145.stderr: 2020-03-11T22:36:55.764 INFO:tasks.workunit.client.0.smithi145.stderr:HEAD is now at 4acc1121e4 Over 30 seconds monitor pg states 2020-03-11T22:36:55.766 INFO:teuthology.orchestra.run.smithi145:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2020-03-11T22:36:55.840 DEBUG:teuthology.orchestra.remote:smithi145:/home/ubuntu/cephtest/workunits.list.client.0 is 1KB 2020-03-11T22:36:55.889 INFO:tasks.workunit:Running workunits matching osd on client.0... 2020-03-11T22:36:55.890 INFO:tasks.workunit:Running workunit osd/divergent-priors.sh... 2020-03-11T22:36:55.890 INFO:teuthology.orchestra.run.smithi145:workunit test osd/divergent-priors.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4acc1121e47039a3cdf2978ae458c10d153aa141 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh 2020-03-11T22:36:55.915 INFO:tasks.workunit.client.0.smithi145.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-03-11T22:36:55.927 INFO:tasks.workunit.client.0.smithi145.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-03-11T22:36:55.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:36:55.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:36:55.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: export PYTHONWARNINGS=ignore 2020-03-11T22:36:55.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: PYTHONWARNINGS=ignore 2020-03-11T22:36:55.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: export CEPH_CONF=/dev/null 2020-03-11T22:36:55.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: CEPH_CONF=/dev/null 2020-03-11T22:36:55.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: main: unset CEPH_ARGS 2020-03-11T22:36:55.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: local code 2020-03-11T22:36:55.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: run td/divergent-priors 2020-03-11T22:36:55.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:21: run: local dir=td/divergent-priors 2020-03-11T22:36:55.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:22: run: shift 2020-03-11T22:36:55.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: export loglen=12 2020-03-11T22:36:55.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: loglen=12 2020-03-11T22:36:55.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: export divisor=3 2020-03-11T22:36:55.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: divisor=3 2020-03-11T22:36:55.931 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: expr 12 / 2 2020-03-11T22:36:55.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: export trim=6 2020-03-11T22:36:55.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: trim=6 2020-03-11T22:36:55.932 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: expr 6 / 3 2020-03-11T22:36:55.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: export DIVERGENT_WRITE=2 2020-03-11T22:36:55.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: DIVERGENT_WRITE=2 2020-03-11T22:36:55.932 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: expr 6 / 3 2020-03-11T22:36:55.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: export DIVERGENT_REMOVE=2 2020-03-11T22:36:55.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: DIVERGENT_REMOVE=2 2020-03-11T22:36:55.933 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: expr 6 / 3 2020-03-11T22:36:55.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: export DIVERGENT_CREATE=2 2020-03-11T22:36:55.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: DIVERGENT_CREATE=2 2020-03-11T22:36:55.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: export poolname=test 2020-03-11T22:36:55.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: poolname=test 2020-03-11T22:36:55.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: export testobjects=100 2020-03-11T22:36:55.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: testobjects=100 2020-03-11T22:36:55.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: export CEPH_MON=127.0.0.1:7115 2020-03-11T22:36:55.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: CEPH_MON=127.0.0.1:7115 2020-03-11T22:36:55.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:35: run: export CEPH_ARGS 2020-03-11T22:36:55.935 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: uuidgen 2020-03-11T22:36:55.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: CEPH_ARGS+='--fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none ' 2020-03-11T22:36:55.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:37: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2020-03-11T22:36:55.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:39: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2020-03-11T22:36:55.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:40: run: CEPH_ARGS+='--osd_debug_pg_log_writeout=true ' 2020-03-11T22:36:55.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:41: run: CEPH_ARGS+='--osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:36:55.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: set 2020-03-11T22:36:55.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-03-11T22:36:55.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: local 'funcs=TEST_divergent 2020-03-11T22:36:55.949 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_divergent_2 2020-03-11T22:36:55.950 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_divergent_3 2020-03-11T22:36:55.950 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_divergent_ec' 2020-03-11T22:36:55.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2020-03-11T22:36:55.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2020-03-11T22:36:55.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2020-03-11T22:36:55.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2020-03-11T22:36:55.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:36:55.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:36:55.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:36:55.951 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:36:55.951 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:36:55.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:36:55.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:36:55.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:36:55.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:36:55.952 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:36:55.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:36:55.952 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:36:55.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:36:55.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:36:55.952 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:36:55.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:36:55.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:36:55.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:36:55.953 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:36:55.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:36:55.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:36:55.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:36:55.954 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:36:55.954 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:55.954 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:36:55.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17402 2020-03-11T22:36:55.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:36:55.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:36:55.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2020-03-11T22:36:55.954 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:36:55.955 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:55.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:36:55.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17402 2020-03-11T22:36:55.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:36:55.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-03-11T22:36:55.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-03-11T22:36:55.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:36:55.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2020-03-11T22:36:55.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent td/divergent-priors 2020-03-11T22:36:55.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:57: TEST_divergent: local dir=td/divergent-priors 2020-03-11T22:36:55.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:60: TEST_divergent: local dummyfile=/etc/fstab 2020-03-11T22:36:55.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:61: TEST_divergent: local dummyfile2=/etc/resolv.conf 2020-03-11T22:36:55.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:63: TEST_divergent: local num_osds=3 2020-03-11T22:36:55.957 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: expr 3 - 1 2020-03-11T22:36:55.957 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: seq 0 2 2020-03-11T22:36:55.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: local 'osds=0 2020-03-11T22:36:55.959 INFO:tasks.workunit.client.0.smithi145.stderr:1 2020-03-11T22:36:55.959 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2020-03-11T22:36:55.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:65: TEST_divergent: run_mon td/divergent-priors a 2020-03-11T22:36:55.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2020-03-11T22:36:55.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:36:55.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:36:55.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:36:55.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2020-03-11T22:36:55.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2020-03-11T22:36:56.375 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:36:56.376 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:36:56.376 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:36:56.376 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:36:56.376 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:56.376 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:36:56.376 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:36:56.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:36:56.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:36:56.521 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:36:56.522 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:36:56.522 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:36:56.522 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:36:56.522 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:36:56.522 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:36:56.522 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:36:56.522 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:36:56.523 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:36:56.523 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:56.523 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:36:56.523 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17402/ceph-mon.a.asok 2020-03-11T22:36:56.523 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:36:56.523 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17402/ceph-mon.a.asok config get fsid 2020-03-11T22:36:56.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:36:56.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:36:56.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:36:56.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:36:56.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:36:56.609 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:36:56.609 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:36:56.609 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:36:56.609 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:36:56.609 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:56.609 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:36:56.609 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17402/ceph-mon.a.asok 2020-03-11T22:36:56.610 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:36:56.610 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17402/ceph-mon.a.asok config get mon_host 2020-03-11T22:36:56.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:66: TEST_divergent: run_mgr td/divergent-priors x 2020-03-11T22:36:56.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/divergent-priors 2020-03-11T22:36:56.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:36:56.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:36:56.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:36:56.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/divergent-priors/x 2020-03-11T22:36:56.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:36:57.600 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:36:57.600 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:36:57.601 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:36:57.601 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:36:57.601 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:57.601 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:36:57.602 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:36:57.602 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:36:57.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:36:57.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2020-03-11T22:36:57.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 0 2020-03-11T22:36:57.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:36:57.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:36:57.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:36:57.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:36:57.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:36:57.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:36:57.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:36:57.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:36:57.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:36:57.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:36:57.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:36:57.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:36:57.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:36:57.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:36:57.635 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:36:57.637 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:36:57.637 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:36:57.638 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:36:57.639 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:57.639 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:36:57.640 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:36:57.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:36:57.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:36:57.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:36:57.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:36:57.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:36:57.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:36:57.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:36:57.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:36:57.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:36:57.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:36:57.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:36:57.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:36:57.643 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:36:57.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=85c0e5de-f314-4f60-acc1-ba357f36e17a 2020-03-11T22:36:57.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 85c0e5de-f314-4f60-acc1-ba357f36e17a' 2020-03-11T22:36:57.644 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:36:57.644 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 85c0e5de-f314-4f60-acc1-ba357f36e17a 2020-03-11T22:36:57.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAJaGleKOt4JhAA7aosqSz3N9lYhkUDwva9Xw== 2020-03-11T22:36:57.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAJaGleKOt4JhAA7aosqSz3N9lYhkUDwva9Xw=="}' 2020-03-11T22:36:57.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 85c0e5de-f314-4f60-acc1-ba357f36e17a -i td/divergent-priors/0/new.json 2020-03-11T22:36:57.982 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:36:57.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/0/new.json 2020-03-11T22:36:57.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAJaGleKOt4JhAA7aosqSz3N9lYhkUDwva9Xw== --osd-uuid 85c0e5de-f314-4f60-acc1-ba357f36e17a 2020-03-11T22:36:58.026 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:36:58.019+0000 7f024d87fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:58.026 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:36:58.021+0000 7f024d87fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:58.026 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:36:58.022+0000 7f024d87fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:58.026 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:36:58.022+0000 7f024d87fec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:36:58.026 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:36:58.022+0000 7f024d87fec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:36:58.027 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:36:58.022+0000 7f024d87fec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:36:58.059 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:36:58.057+0000 7f024d87fec0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2020-03-11T22:37:01.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/0/keyring 2020-03-11T22:37:01.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:37:01.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:37:01.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:37:01.032 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2020-03-11T22:37:01.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:37:01.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:37:01.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:37:01.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:37:01.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:37:01.316 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2020-03-11T22:37:01.344 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:37:01.341+0000 7f68f88aaec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:01.356 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:37:01.354+0000 7f68f88aaec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:01.357 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:37:01.355+0000 7f68f88aaec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:01.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:37:01.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:37:01.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:37:01.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:37:01.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:37:01.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:01.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:37:01.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:01.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:37:01.586 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:37:01.873 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:37:01.870+0000 7f68f88aaec0 -1 Falling back to public interface 2020-03-11T22:37:01.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:37:02.673 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:37:02.669+0000 7f68f88aaec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:37:02.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:37:02.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:02.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:37:02.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:37:02.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:02.934 INFO:tasks.workunit.client.0.smithi145.stdout:1 2020-03-11T22:37:03.329 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/17784,v1:127.0.0.1:6803/17784] [v2:127.0.0.1:6804/17784,v1:127.0.0.1:6805/17784] exists,up 85c0e5de-f314-4f60-acc1-ba357f36e17a 2020-03-11T22:37:03.329 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 29aaf178-2506-493e-b1ca-ca8045da325e 2020-03-11T22:37:03.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:37:03.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:37:03.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:37:03.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2020-03-11T22:37:03.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 1 2020-03-11T22:37:03.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:37:03.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:37:03.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:37:03.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:37:03.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:37:03.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:37:03.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:37:03.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:37:03.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:37:03.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:37:03.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:37:03.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:37:03.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:37:03.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:37:03.332 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:37:03.332 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:37:03.332 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:37:03.333 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:37:03.333 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:03.333 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:37:03.333 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:37:03.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:37:03.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:37:03.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:37:03.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:37:03.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:37:03.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:37:03.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:37:03.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:37:03.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:37:03.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:37:03.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:37:03.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:37:03.335 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:37:03.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=29aaf178-2506-493e-b1ca-ca8045da325e 2020-03-11T22:37:03.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 29aaf178-2506-493e-b1ca-ca8045da325e' 2020-03-11T22:37:03.335 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:37:03.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAPaGle73iMFBAAvEt2bmUjsnuFKLoGzpFtvg== 2020-03-11T22:37:03.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAPaGle73iMFBAAvEt2bmUjsnuFKLoGzpFtvg=="}' 2020-03-11T22:37:03.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 29aaf178-2506-493e-b1ca-ca8045da325e -i td/divergent-priors/1/new.json 2020-03-11T22:37:03.821 INFO:tasks.workunit.client.0.smithi145.stdout:1 2020-03-11T22:37:03.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/1/new.json 2020-03-11T22:37:03.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAPaGle73iMFBAAvEt2bmUjsnuFKLoGzpFtvg== --osd-uuid 29aaf178-2506-493e-b1ca-ca8045da325e 2020-03-11T22:37:03.861 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:37:03.858+0000 7f3c3cba1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:03.870 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:37:03.868+0000 7f3c3cba1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:03.872 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:37:03.870+0000 7f3c3cba1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:03.873 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:37:03.870+0000 7f3c3cba1ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:37:03.873 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:37:03.870+0000 7f3c3cba1ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:37:03.873 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:37:03.870+0000 7f3c3cba1ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:37:03.929 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:37:03.926+0000 7f3c3cba1ec0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2020-03-11T22:37:06.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/1/keyring 2020-03-11T22:37:06.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:37:06.905 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2020-03-11T22:37:06.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:37:06.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:37:07.363 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2020-03-11T22:37:07.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:37:07.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:37:07.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:37:07.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:37:07.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:37:07.390 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:37:07.387+0000 7f878327dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:07.392 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:37:07.389+0000 7f878327dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:07.394 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:37:07.391+0000 7f878327dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:07.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:37:07.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:37:07.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:37:07.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:37:07.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:37:07.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:07.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:37:07.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:07.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:37:07.729 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:37:07.912 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:37:07.910+0000 7f878327dec0 -1 Falling back to public interface 2020-03-11T22:37:08.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:37:08.618 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:37:08.615+0000 7f878327dec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:37:09.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:37:09.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:09.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:37:09.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:09.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:37:09.077 INFO:tasks.workunit.client.0.smithi145.stdout:1 2020-03-11T22:37:09.425 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/18194,v1:127.0.0.1:6811/18194] [v2:127.0.0.1:6812/18194,v1:127.0.0.1:6813/18194] exists,up 29aaf178-2506-493e-b1ca-ca8045da325e 2020-03-11T22:37:09.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:37:09.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:37:09.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:37:09.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2020-03-11T22:37:09.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 2 2020-03-11T22:37:09.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:37:09.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:37:09.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:37:09.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:37:09.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:37:09.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:37:09.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:37:09.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:37:09.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:37:09.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:37:09.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:37:09.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:37:09.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:37:09.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:37:09.428 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:37:09.428 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:37:09.428 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:37:09.429 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:37:09.429 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:09.429 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:37:09.429 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:37:09.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:37:09.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:37:09.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:37:09.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:37:09.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:37:09.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:37:09.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:37:09.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:37:09.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:37:09.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:37:09.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:37:09.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:37:09.431 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:37:09.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=52ae9eef-8cbf-42f9-836d-cf45b3fb4952 2020-03-11T22:37:09.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 52ae9eef-8cbf-42f9-836d-cf45b3fb4952' 2020-03-11T22:37:09.432 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:37:09.432 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 52ae9eef-8cbf-42f9-836d-cf45b3fb4952 2020-03-11T22:37:09.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAVaGleMuFhGhAAArVw1m7i1XEejl3HYvayrQ== 2020-03-11T22:37:09.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAVaGleMuFhGhAAArVw1m7i1XEejl3HYvayrQ=="}' 2020-03-11T22:37:09.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 52ae9eef-8cbf-42f9-836d-cf45b3fb4952 -i td/divergent-priors/2/new.json 2020-03-11T22:37:09.948 INFO:tasks.workunit.client.0.smithi145.stdout:2 2020-03-11T22:37:09.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/2/new.json 2020-03-11T22:37:09.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAVaGleMuFhGhAAArVw1m7i1XEejl3HYvayrQ== --osd-uuid 52ae9eef-8cbf-42f9-836d-cf45b3fb4952 2020-03-11T22:37:09.987 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:37:09.984+0000 7f571151eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:09.988 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:37:09.986+0000 7f571151eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:09.990 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:37:09.987+0000 7f571151eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:09.991 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:37:09.988+0000 7f571151eec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:37:09.991 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:37:09.988+0000 7f571151eec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:37:09.991 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:37:09.988+0000 7f571151eec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:37:10.050 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:37:10.047+0000 7f571151eec0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2020-03-11T22:37:13.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/2/keyring 2020-03-11T22:37:13.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:37:13.185 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2020-03-11T22:37:13.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:37:13.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:37:13.645 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2020-03-11T22:37:13.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:37:13.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:37:13.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:37:13.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:37:13.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:37:13.671 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:37:13.668+0000 7f19226e0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:13.673 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:37:13.670+0000 7f19226e0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:13.674 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:37:13.671+0000 7f19226e0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:14.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:37:14.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:37:14.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:37:14.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:37:14.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:37:14.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:14.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:37:14.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:14.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:37:14.023 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:37:14.189 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:37:14.185+0000 7f19226e0ec0 -1 Falling back to public interface 2020-03-11T22:37:14.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:37:14.923 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:37:14.920+0000 7f19226e0ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:37:15.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:37:15.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:15.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:37:15.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:37:15.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:15.394 INFO:tasks.workunit.client.0.smithi145.stdout:1 2020-03-11T22:37:15.751 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/18557,v1:127.0.0.1:6819/18557] [v2:127.0.0.1:6820/18557,v1:127.0.0.1:6821/18557] exists,up 52ae9eef-8cbf-42f9-836d-cf45b3fb4952 2020-03-11T22:37:15.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:37:15.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:37:15.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:37:15.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:72: TEST_divergent: ceph osd set noout 2020-03-11T22:37:16.442 INFO:tasks.workunit.client.0.smithi145.stderr:noout is set 2020-03-11T22:37:16.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:73: TEST_divergent: ceph osd set noin 2020-03-11T22:37:17.026 INFO:tasks.workunit.client.0.smithi145.stderr:noin is set 2020-03-11T22:37:17.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:74: TEST_divergent: ceph osd set nodown 2020-03-11T22:37:17.665 INFO:tasks.workunit.client.0.smithi145.stderr:nodown is set 2020-03-11T22:37:17.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:75: TEST_divergent: create_pool test 1 1 2020-03-11T22:37:17.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-03-11T22:37:18.238 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' already exists 2020-03-11T22:37:18.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:37:19.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:76: TEST_divergent: ceph osd pool set test size 3 2020-03-11T22:37:19.957 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 3 2020-03-11T22:37:19.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:77: TEST_divergent: ceph osd pool set test min_size 2 2020-03-11T22:37:20.632 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 min_size to 2 2020-03-11T22:37:20.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:79: TEST_divergent: flush_pg_stats 2020-03-11T22:37:20.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:37:20.641 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:37:20.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:37:20.987 INFO:tasks.workunit.client.0.smithi145.stderr:1 2020-03-11T22:37:20.987 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2020-03-11T22:37:20.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:37:20.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:37:20.988 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:37:21.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836486 2020-03-11T22:37:21.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486' 2020-03-11T22:37:21.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:37:21.117 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:37:21.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672966 2020-03-11T22:37:21.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672966' 2020-03-11T22:37:21.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:37:21.248 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:37:21.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509444 2020-03-11T22:37:21.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672966 2-64424509444' 2020-03-11T22:37:21.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:37:21.378 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836486 2020-03-11T22:37:21.378 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:37:21.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:37:21.379 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836486 2020-03-11T22:37:21.380 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:37:21.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836486 2020-03-11T22:37:21.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2020-03-11T22:37:21.381 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836486 2020-03-11T22:37:21.381 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:37:21.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836486 -lt 21474836486 2020-03-11T22:37:21.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:37:21.730 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672966 2020-03-11T22:37:21.730 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:37:21.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:37:21.732 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672966 2020-03-11T22:37:21.732 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:37:21.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672966 2020-03-11T22:37:21.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2020-03-11T22:37:21.733 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672966 2020-03-11T22:37:21.734 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:37:22.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672966 -lt 42949672966 2020-03-11T22:37:22.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:37:22.077 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509444 2020-03-11T22:37:22.077 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:37:22.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:37:22.079 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509444 2020-03-11T22:37:22.079 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:37:22.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509444 2020-03-11T22:37:22.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2020-03-11T22:37:22.081 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509444 2020-03-11T22:37:22.081 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:37:22.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509444 -lt 64424509444 2020-03-11T22:37:22.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:80: TEST_divergent: wait_for_clean 2020-03-11T22:37:22.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:37:22.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:37:22.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:37:22.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:37:22.423 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:37:22.423 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:37:22.423 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:37:22.423 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:37:22.423 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:37:22.423 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:37:22.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:37:22.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:37:22.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:37:22.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:37:22.513 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:37:22.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:37:22.858 INFO:tasks.workunit.client.0.smithi145.stderr:1 2020-03-11T22:37:22.858 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2020-03-11T22:37:22.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:37:22.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:37:22.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:37:22.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836488 2020-03-11T22:37:22.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836488' 2020-03-11T22:37:22.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:37:22.988 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:37:23.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672968 2020-03-11T22:37:23.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836488 1-42949672968' 2020-03-11T22:37:23.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:37:23.117 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:37:23.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509446 2020-03-11T22:37:23.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836488 1-42949672968 2-64424509446' 2020-03-11T22:37:23.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:37:23.247 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836488 2020-03-11T22:37:23.248 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:37:23.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:37:23.250 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836488 2020-03-11T22:37:23.250 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:37:23.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836488 2020-03-11T22:37:23.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2020-03-11T22:37:23.251 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836488 2020-03-11T22:37:23.251 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:37:23.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836488 -lt 21474836488 2020-03-11T22:37:23.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:37:23.602 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672968 2020-03-11T22:37:23.602 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:37:23.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:37:23.604 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672968 2020-03-11T22:37:23.604 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:37:23.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672968 2020-03-11T22:37:23.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2020-03-11T22:37:23.605 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672968 2020-03-11T22:37:23.605 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:37:23.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672968 -lt 42949672968 2020-03-11T22:37:23.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:37:23.948 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509446 2020-03-11T22:37:23.948 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:37:23.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:37:23.950 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509446 2020-03-11T22:37:23.950 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:37:23.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509446 2020-03-11T22:37:23.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2020-03-11T22:37:23.951 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509446 2020-03-11T22:37:23.952 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:37:24.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509446 -lt 64424509446 2020-03-11T22:37:24.305 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:37:24.305 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:24.305 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:24.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:37:24.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:24.749 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:24.749 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:24.749 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:24.749 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:24.749 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:24.749 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:25.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:37:25.081 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:25.081 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:25.081 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:25.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:37:25.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:37:25.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:37:25.522 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: ceph pg dump pgs --format=json 2020-03-11T22:37:25.522 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: jq '.pg_stats[0].up_primary' 2020-03-11T22:37:25.846 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:37:25.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: local divergent=1 2020-03-11T22:37:25.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:84: TEST_divergent: echo 'primary and soon to be divergent is 1' 2020-03-11T22:37:25.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:85: TEST_divergent: ceph pg dump pgs 2020-03-11T22:37:25.855 INFO:tasks.workunit.client.0.smithi145.stdout:primary and soon to be divergent is 1 2020-03-11T22:37:26.173 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:37:26.173 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:37:18.983458+0000 0'0 28:12 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:37:17.979060+0000 0'0 2020-03-11T22:37:17.979060+0000 0 2020-03-11T22:37:26.174 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:37:26.174 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:37:26.175 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:37:26.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:86: TEST_divergent: local non_divergent= 2020-03-11T22:37:26.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2020-03-11T22:37:26.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 0 = 1 ']' 2020-03-11T22:37:26.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0' 2020-03-11T22:37:26.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2020-03-11T22:37:26.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 1 = 1 ']' 2020-03-11T22:37:26.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:90: TEST_divergent: continue 2020-03-11T22:37:26.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2020-03-11T22:37:26.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 2 = 1 ']' 2020-03-11T22:37:26.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0 2' 2020-03-11T22:37:26.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:95: TEST_divergent: echo 'writing initial objects' 2020-03-11T22:37:26.185 INFO:tasks.workunit.client.0.smithi145.stdout:writing initial objects 2020-03-11T22:37:26.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: seq 1 100 2020-03-11T22:37:26.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:26.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_1 /etc/fstab 2020-03-11T22:37:26.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:26.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_2 /etc/fstab 2020-03-11T22:37:26.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:26.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_3 /etc/fstab 2020-03-11T22:37:26.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:26.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_4 /etc/fstab 2020-03-11T22:37:26.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:26.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_5 /etc/fstab 2020-03-11T22:37:26.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:26.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_6 /etc/fstab 2020-03-11T22:37:26.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:26.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_7 /etc/fstab 2020-03-11T22:37:26.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:26.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_8 /etc/fstab 2020-03-11T22:37:26.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:26.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_9 /etc/fstab 2020-03-11T22:37:26.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:26.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_10 /etc/fstab 2020-03-11T22:37:26.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:26.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_11 /etc/fstab 2020-03-11T22:37:26.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:26.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_12 /etc/fstab 2020-03-11T22:37:26.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:26.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_13 /etc/fstab 2020-03-11T22:37:26.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:26.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_14 /etc/fstab 2020-03-11T22:37:26.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:26.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_15 /etc/fstab 2020-03-11T22:37:26.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:26.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_16 /etc/fstab 2020-03-11T22:37:27.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:27.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_17 /etc/fstab 2020-03-11T22:37:27.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:27.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_18 /etc/fstab 2020-03-11T22:37:27.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:27.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_19 /etc/fstab 2020-03-11T22:37:27.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:27.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_20 /etc/fstab 2020-03-11T22:37:27.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:27.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_21 /etc/fstab 2020-03-11T22:37:27.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:27.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_22 /etc/fstab 2020-03-11T22:37:27.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:27.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_23 /etc/fstab 2020-03-11T22:37:27.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:27.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_24 /etc/fstab 2020-03-11T22:37:27.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:27.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_25 /etc/fstab 2020-03-11T22:37:27.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:27.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_26 /etc/fstab 2020-03-11T22:37:27.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:27.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_27 /etc/fstab 2020-03-11T22:37:27.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:27.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_28 /etc/fstab 2020-03-11T22:37:27.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:27.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_29 /etc/fstab 2020-03-11T22:37:27.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:27.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_30 /etc/fstab 2020-03-11T22:37:27.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:27.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_31 /etc/fstab 2020-03-11T22:37:27.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:27.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_32 /etc/fstab 2020-03-11T22:37:28.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:28.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_33 /etc/fstab 2020-03-11T22:37:28.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:28.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_34 /etc/fstab 2020-03-11T22:37:28.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:28.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_35 /etc/fstab 2020-03-11T22:37:28.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:28.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_36 /etc/fstab 2020-03-11T22:37:29.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:29.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_37 /etc/fstab 2020-03-11T22:37:29.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:29.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_38 /etc/fstab 2020-03-11T22:37:29.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:29.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_39 /etc/fstab 2020-03-11T22:37:29.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:29.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_40 /etc/fstab 2020-03-11T22:37:29.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:29.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_41 /etc/fstab 2020-03-11T22:37:30.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:30.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_42 /etc/fstab 2020-03-11T22:37:30.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:30.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_43 /etc/fstab 2020-03-11T22:37:30.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:30.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_44 /etc/fstab 2020-03-11T22:37:30.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:30.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_45 /etc/fstab 2020-03-11T22:37:30.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:30.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_46 /etc/fstab 2020-03-11T22:37:30.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:30.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_47 /etc/fstab 2020-03-11T22:37:30.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:30.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_48 /etc/fstab 2020-03-11T22:37:30.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:30.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_49 /etc/fstab 2020-03-11T22:37:30.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:30.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_50 /etc/fstab 2020-03-11T22:37:30.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:30.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_51 /etc/fstab 2020-03-11T22:37:30.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:30.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_52 /etc/fstab 2020-03-11T22:37:30.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:30.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_53 /etc/fstab 2020-03-11T22:37:30.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:30.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_54 /etc/fstab 2020-03-11T22:37:30.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:30.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_55 /etc/fstab 2020-03-11T22:37:31.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:31.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_56 /etc/fstab 2020-03-11T22:37:31.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:31.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_57 /etc/fstab 2020-03-11T22:37:31.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:31.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_58 /etc/fstab 2020-03-11T22:37:31.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:31.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_59 /etc/fstab 2020-03-11T22:37:31.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:31.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_60 /etc/fstab 2020-03-11T22:37:31.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:31.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_61 /etc/fstab 2020-03-11T22:37:31.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:31.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_62 /etc/fstab 2020-03-11T22:37:31.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:31.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_63 /etc/fstab 2020-03-11T22:37:31.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:31.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_64 /etc/fstab 2020-03-11T22:37:31.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:31.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_65 /etc/fstab 2020-03-11T22:37:31.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:31.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_66 /etc/fstab 2020-03-11T22:37:31.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:31.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_67 /etc/fstab 2020-03-11T22:37:31.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:31.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_68 /etc/fstab 2020-03-11T22:37:31.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:31.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_69 /etc/fstab 2020-03-11T22:37:31.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:31.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_70 /etc/fstab 2020-03-11T22:37:31.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:31.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_71 /etc/fstab 2020-03-11T22:37:31.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:31.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_72 /etc/fstab 2020-03-11T22:37:31.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:31.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_73 /etc/fstab 2020-03-11T22:37:32.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:32.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_74 /etc/fstab 2020-03-11T22:37:32.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:32.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_75 /etc/fstab 2020-03-11T22:37:32.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:32.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_76 /etc/fstab 2020-03-11T22:37:32.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:32.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_77 /etc/fstab 2020-03-11T22:37:32.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:32.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_78 /etc/fstab 2020-03-11T22:37:32.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:32.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_79 /etc/fstab 2020-03-11T22:37:32.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:32.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_80 /etc/fstab 2020-03-11T22:37:32.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:32.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_81 /etc/fstab 2020-03-11T22:37:32.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:32.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_82 /etc/fstab 2020-03-11T22:37:32.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:32.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_83 /etc/fstab 2020-03-11T22:37:32.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:32.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_84 /etc/fstab 2020-03-11T22:37:32.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:32.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_85 /etc/fstab 2020-03-11T22:37:32.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:32.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_86 /etc/fstab 2020-03-11T22:37:32.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:32.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_87 /etc/fstab 2020-03-11T22:37:32.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:32.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_88 /etc/fstab 2020-03-11T22:37:32.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:32.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_89 /etc/fstab 2020-03-11T22:37:32.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:32.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_90 /etc/fstab 2020-03-11T22:37:32.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:32.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_91 /etc/fstab 2020-03-11T22:37:32.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:32.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_92 /etc/fstab 2020-03-11T22:37:33.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:33.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_93 /etc/fstab 2020-03-11T22:37:33.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:33.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_94 /etc/fstab 2020-03-11T22:37:33.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:33.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_95 /etc/fstab 2020-03-11T22:37:33.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:33.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_96 /etc/fstab 2020-03-11T22:37:33.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:33.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_97 /etc/fstab 2020-03-11T22:37:33.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:33.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_98 /etc/fstab 2020-03-11T22:37:33.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:33.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_99 /etc/fstab 2020-03-11T22:37:33.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:37:33.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_100 /etc/fstab 2020-03-11T22:37:33.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:37:33.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: wait_for_clean 2020-03-11T22:37:33.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:37:33.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:37:33.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:37:33.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:37:33.504 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:37:33.504 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:37:33.504 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:37:33.504 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:37:33.504 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:37:33.505 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:37:33.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:37:33.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:37:33.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:37:33.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:37:33.559 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:37:33.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:37:33.903 INFO:tasks.workunit.client.0.smithi145.stderr:1 2020-03-11T22:37:33.904 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2020-03-11T22:37:33.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:37:33.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:37:33.904 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:37:34.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836493 2020-03-11T22:37:34.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493' 2020-03-11T22:37:34.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:37:34.034 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:37:34.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672972 2020-03-11T22:37:34.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672972' 2020-03-11T22:37:34.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:37:34.164 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:37:34.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509450 2020-03-11T22:37:34.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672972 2-64424509450' 2020-03-11T22:37:34.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:37:34.292 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836493 2020-03-11T22:37:34.292 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:37:34.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:37:34.294 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836493 2020-03-11T22:37:34.294 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:37:34.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836493 2020-03-11T22:37:34.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2020-03-11T22:37:34.295 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836493 2020-03-11T22:37:34.296 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:37:34.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836490 -lt 21474836493 2020-03-11T22:37:34.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:37:35.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:37:35.639 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:37:35.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836493 -lt 21474836493 2020-03-11T22:37:35.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:37:35.989 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672972 2020-03-11T22:37:35.990 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:37:35.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:37:35.991 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672972 2020-03-11T22:37:35.992 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:37:35.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672972 2020-03-11T22:37:35.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2020-03-11T22:37:35.993 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672972 2020-03-11T22:37:35.993 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:37:36.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672973 -lt 42949672972 2020-03-11T22:37:36.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:37:36.339 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509450 2020-03-11T22:37:36.340 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:37:36.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:37:36.341 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509450 2020-03-11T22:37:36.341 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:37:36.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509450 2020-03-11T22:37:36.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2020-03-11T22:37:36.343 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509450 2020-03-11T22:37:36.343 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:37:36.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509450 -lt 64424509450 2020-03-11T22:37:36.696 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:37:36.697 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:36.697 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:37.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:37:37.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:37:37.139 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:37:37.140 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:37:37.140 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:37:37.140 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:37:37.140 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:37:37.140 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:37:37.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:37:37.467 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:37:37.468 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:37.468 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:37:37.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:37:37.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:37:37.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:37:37.910 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: get_pg test existing_1 2020-03-11T22:37:37.910 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local poolname=test 2020-03-11T22:37:37.910 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local objectname=existing_1 2020-03-11T22:37:37.910 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: ceph --format json osd map test existing_1 2020-03-11T22:37:37.910 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: jq -r .pgid 2020-03-11T22:37:38.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: local pgid=1.0 2020-03-11T22:37:38.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:107: TEST_divergent: echo 'blackholing osds 0 2' 2020-03-11T22:37:38.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:108: TEST_divergent: ceph pg dump pgs 2020-03-11T22:37:38.270 INFO:tasks.workunit.client.0.smithi145.stdout:blackholing osds 0 2 2020-03-11T22:37:38.590 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:37:38.590 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2020-03-11T22:37:18.983458+0000 29'100 29:112 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:37:17.979060+0000 0'0 2020-03-11T22:37:17.979060+0000 0 2020-03-11T22:37:38.590 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:37:38.590 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:37:38.591 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:37:38.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2020-03-11T22:37:38.601 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.0 2020-03-11T22:37:38.601 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:37:38.601 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:37:38.601 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:37:38.601 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:38.601 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:37:38.602 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17402/ceph-osd.0.asok 2020-03-11T22:37:38.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2020-03-11T22:37:38.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.17402/ceph-osd.0.asok config set objectstore_blackhole 1 2020-03-11T22:37:38.686 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:37:38.687 INFO:tasks.workunit.client.0.smithi145.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:37:38.687 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:37:38.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2020-03-11T22:37:38.695 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.2 2020-03-11T22:37:38.695 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:37:38.695 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:37:38.695 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:37:38.695 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:38.696 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:37:38.696 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17402/ceph-osd.2.asok 2020-03-11T22:37:38.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2020-03-11T22:37:38.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.17402/ceph-osd.2.asok config set objectstore_blackhole 1 2020-03-11T22:37:38.782 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:37:38.782 INFO:tasks.workunit.client.0.smithi145.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:37:38.782 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:37:38.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:114: TEST_divergent: local case5=100 2020-03-11T22:37:38.789 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: expr 100 - 1 2020-03-11T22:37:38.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: local case3=99 2020-03-11T22:37:38.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:117: TEST_divergent: echo 'writing divergent object' 2020-03-11T22:37:38.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:119: TEST_divergent: echo 'create missing divergent object' 2020-03-11T22:37:38.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:120: TEST_divergent: inject_eio rep data test existing_99 td/divergent-priors 0 2020-03-11T22:37:38.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:118: TEST_divergent: rados -p test put existing_100 /etc/fstab 2020-03-11T22:37:38.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: inject_eio: local pooltype=rep 2020-03-11T22:37:38.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: inject_eio: shift 2020-03-11T22:37:38.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: inject_eio: local which=data 2020-03-11T22:37:38.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: inject_eio: shift 2020-03-11T22:37:38.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: inject_eio: local poolname=test 2020-03-11T22:37:38.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: shift 2020-03-11T22:37:38.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: local objname=existing_99 2020-03-11T22:37:38.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: shift 2020-03-11T22:37:38.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: local dir=td/divergent-priors 2020-03-11T22:37:38.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: shift 2020-03-11T22:37:38.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: local shard_id=0 2020-03-11T22:37:38.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: shift 2020-03-11T22:37:38.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2020-03-11T22:37:38.795 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: get_osds test existing_99 2020-03-11T22:37:38.795 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: get_osds: local poolname=test 2020-03-11T22:37:38.795 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local objectname=existing_99 2020-03-11T22:37:38.795 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: get_osds: ceph --format json osd map test existing_99 2020-03-11T22:37:38.795 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: get_osds: jq '.acting | .[]' 2020-03-11T22:37:38.795 INFO:tasks.workunit.client.0.smithi145.stdout:writing divergent object 2020-03-11T22:37:38.796 INFO:tasks.workunit.client.0.smithi145.stdout:create missing divergent object 2020-03-11T22:37:39.147 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: get_osds: local 'osds=1 2020-03-11T22:37:39.147 INFO:tasks.workunit.client.0.smithi145.stderr:0 2020-03-11T22:37:39.147 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2020-03-11T22:37:39.147 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:964: get_osds: echo 1 0 2 2020-03-11T22:37:39.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local -a initial_osds 2020-03-11T22:37:39.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: local osd_id=1 2020-03-11T22:37:39.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: '[' rep '!=' ec ']' 2020-03-11T22:37:39.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shard_id= 2020-03-11T22:37:39.149 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: cat td/divergent-priors/1/type 2020-03-11T22:37:39.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: type=bluestore 2020-03-11T22:37:39.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2020-03-11T22:37:39.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1098: set_config: local daemon=osd 2020-03-11T22:37:39.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local id=1 2020-03-11T22:37:39.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local config=bluestore_debug_inject_read_err 2020-03-11T22:37:39.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local value=true 2020-03-11T22:37:39.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: jq 'has("success")' 2020-03-11T22:37:39.151 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: get_asok_path osd.1 2020-03-11T22:37:39.151 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:37:39.151 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:37:39.151 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:37:39.151 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:39.152 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:37:39.152 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17402/ceph-osd.1.asok 2020-03-11T22:37:39.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.17402/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2020-03-11T22:37:39.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: test true == true 2020-03-11T22:37:39.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: local loop=0 2020-03-11T22:37:39.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: inject_eio: grep -q Invalid 2020-03-11T22:37:39.246 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: get_asok_path osd.1 2020-03-11T22:37:39.246 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:37:39.246 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:37:39.246 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:37:39.247 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:39.247 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:37:39.247 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17402/ceph-osd.1.asok 2020-03-11T22:37:39.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: CEPH_ARGS= 2020-03-11T22:37:39.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: ceph --admin-daemon /tmp/ceph-asok.17402/ceph-osd.1.asok injectdataerr test existing_99 2020-03-11T22:37:39.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:122: TEST_divergent: sleep 10 2020-03-11T22:37:39.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:121: TEST_divergent: rados -p test get existing_99 td/divergent-priors/existing 2020-03-11T22:37:49.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:123: TEST_divergent: killall -9 rados 2020-03-11T22:37:49.362 INFO:tasks.workunit.client.0.smithi145.stderr:rados: no process found 2020-03-11T22:37:49.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:126: TEST_divergent: echo 'killing all the osds' 2020-03-11T22:37:49.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:127: TEST_divergent: ceph pg dump pgs 2020-03-11T22:37:49.363 INFO:tasks.workunit.client.0.smithi145.stdout:killing all the osds 2020-03-11T22:37:49.695 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:37:49.695 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 0 0 0 48300 0 0 17 17 active+clean 2020-03-11T22:37:39.597408+0000 29'101 29:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:37:17.979060+0000 0'0 2020-03-11T22:37:17.979060+0000 0 2020-03-11T22:37:49.695 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:37:49.695 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:37:49.696 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:37:49.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:128: TEST_divergent: kill_daemons td/divergent-priors KILL osd 2020-03-11T22:37:49.706 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:37:49.706 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:37:49.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:37:49.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:37:49.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:37:49.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:37:49.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2020-03-11T22:37:49.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.0 2020-03-11T22:37:50.291 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already down. 2020-03-11T22:37:50.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2020-03-11T22:37:50.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.1 2020-03-11T22:37:50.782 INFO:tasks.workunit.client.0.smithi145.stderr:osd.1 is already down. 2020-03-11T22:37:50.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2020-03-11T22:37:50.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.2 2020-03-11T22:37:51.282 INFO:tasks.workunit.client.0.smithi145.stderr:osd.2 is already down. 2020-03-11T22:37:51.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2020-03-11T22:37:51.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.0 2020-03-11T22:37:51.875 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already out. 2020-03-11T22:37:51.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2020-03-11T22:37:51.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.2 2020-03-11T22:37:52.377 INFO:tasks.workunit.client.0.smithi145.stderr:osd.2 is already out. 2020-03-11T22:37:52.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:139: TEST_divergent: echo 'bringing up non_divergent 0 2' 2020-03-11T22:37:52.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:140: TEST_divergent: ceph pg dump pgs 2020-03-11T22:37:52.385 INFO:tasks.workunit.client.0.smithi145.stdout:bringing up non_divergent 0 2 2020-03-11T22:37:52.703 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:37:52.704 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 0 0 0 48300 0 0 17 17 stale+active+clean 2020-03-11T22:37:39.597408+0000 29'101 29:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:37:17.979060+0000 0'0 2020-03-11T22:37:17.979060+0000 0 2020-03-11T22:37:52.704 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:37:52.704 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:37:52.705 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:37:52.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2020-03-11T22:37:52.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 0 2020-03-11T22:37:52.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:37:52.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:37:52.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:37:52.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:37:52.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:37:52.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:37:52.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:37:52.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:37:52.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:37:52.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:37:52.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:37:52.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:37:52.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:37:52.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:37:52.719 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:37:52.719 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:37:52.720 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:37:52.720 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:37:52.720 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:52.720 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:37:52.720 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:37:52.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:37:52.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:37:52.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:37:52.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:37:52.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:37:52.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:37:52.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:37:52.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:37:52.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:37:52.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:37:52.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:37:52.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:37:52.722 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-03-11T22:37:52.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:37:52.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:37:52.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:37:52.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:37:52.723 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2020-03-11T22:37:52.742 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:37:52.740+0000 7f0b86bedec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:52.744 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:37:52.741+0000 7f0b86bedec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:52.746 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:37:52.743+0000 7f0b86bedec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:53.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:37:53.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:37:53.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:37:53.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:37:53.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:37:53.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:53.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:37:53.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:53.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:37:53.061 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:37:53.262 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:37:53.260+0000 7f0b86bedec0 -1 Falling back to public interface 2020-03-11T22:37:53.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:37:54.088 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:37:54.086+0000 7f0b86bedec0 -1 osd.0 29 log_to_monitors {default=true} 2020-03-11T22:37:54.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:37:54.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:54.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:37:54.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:54.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:37:54.418 INFO:tasks.workunit.client.0.smithi145.stdout:1 2020-03-11T22:37:54.765 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up out weight 0 up_from 35 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/22517,v1:127.0.0.1:6803/22517] [v2:127.0.0.1:6804/22517,v1:127.0.0.1:6805/22517] exists,up 85c0e5de-f314-4f60-acc1-ba357f36e17a 2020-03-11T22:37:54.766 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2020-03-11T22:37:54.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:37:54.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:37:54.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:37:54.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2020-03-11T22:37:54.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 2 2020-03-11T22:37:54.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:37:54.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:37:54.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:37:54.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:37:54.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:37:54.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:37:54.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:37:54.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:37:54.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:37:54.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:37:54.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:37:54.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:37:54.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:37:54.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:37:54.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:37:54.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:37:54.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:37:54.770 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:37:54.771 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:54.771 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:37:54.771 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:37:54.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:37:54.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:37:54.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:37:54.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:37:54.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:37:54.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:37:54.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:37:54.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:37:54.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:37:54.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:37:54.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:37:54.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:37:54.773 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-03-11T22:37:54.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:37:54.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:37:54.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:37:54.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:37:54.790 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:37:54.788+0000 7f48bc6d3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:54.792 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:37:54.789+0000 7f48bc6d3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:54.795 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:37:54.792+0000 7f48bc6d3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:55.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:37:55.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:37:55.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:37:55.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:37:55.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:37:55.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:55.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:37:55.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:37:55.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:55.115 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:37:55.308 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:37:55.305+0000 7f48bc6d3ec0 -1 Falling back to public interface 2020-03-11T22:37:55.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:37:56.158 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:37:56.155+0000 7f48bc6d3ec0 -1 osd.2 29 log_to_monitors {default=true} 2020-03-11T22:37:56.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:37:56.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:56.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:37:56.460 INFO:tasks.workunit.client.0.smithi145.stdout:1 2020-03-11T22:37:56.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:37:56.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:56.814 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up out weight 0 up_from 36 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/22711,v1:127.0.0.1:6811/22711] [v2:127.0.0.1:6812/22711,v1:127.0.0.1:6813/22711] exists,up 52ae9eef-8cbf-42f9-836d-cf45b3fb4952 2020-03-11T22:37:56.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:37:56.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:37:56.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:37:56.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2020-03-11T22:37:56.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.0 2020-03-11T22:37:57.324 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already in. 2020-03-11T22:37:57.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2020-03-11T22:37:57.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.2 2020-03-11T22:37:57.912 INFO:tasks.workunit.client.0.smithi145.stderr:osd.2 is already in. 2020-03-11T22:37:57.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:37:57.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: wait_for_clean 2020-03-11T22:37:57.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:37:57.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:37:57.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:37:57.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:37:57.921 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:37:57.922 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:37:57.922 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:37:57.922 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:37:57.923 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:37:57.923 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:37:57.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:37:57.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:37:57.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:37:57.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:37:57.978 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:37:58.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:37:58.375 INFO:tasks.workunit.client.0.smithi145.stderr:1 2020-03-11T22:37:58.375 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2020-03-11T22:37:58.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:37:58.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:37:58.376 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:37:58.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=150323855363 2020-03-11T22:37:58.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-150323855363' 2020-03-11T22:37:58.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:37:58.506 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:37:58.596 INFO:tasks.workunit.client.0.smithi145.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:37:58.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:37:58.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-150323855363 1-' 2020-03-11T22:37:58.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:37:58.605 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:37:58.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=154618822659 2020-03-11T22:37:58.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-150323855363 1- 2-154618822659' 2020-03-11T22:37:58.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:37:58.734 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-150323855363 2020-03-11T22:37:58.734 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:37:58.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:37:58.735 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-150323855363 2020-03-11T22:37:58.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:37:58.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=150323855363 2020-03-11T22:37:58.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 150323855363' 2020-03-11T22:37:58.737 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:37:58.737 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 150323855363 2020-03-11T22:37:59.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 150323855363 2020-03-11T22:37:59.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:38:00.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:38:00.078 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:38:00.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855364 -lt 150323855363 2020-03-11T22:38:00.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:38:00.428 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:38:00.429 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:38:00.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:38:00.430 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:38:00.430 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:38:00.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:38:00.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:38:00.431 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:38:00.431 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 2020-03-11T22:38:00.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672975 -lt 2020-03-11T22:38:00.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672975: unary operator expected 2020-03-11T22:38:00.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:38:00.775 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-154618822659 2020-03-11T22:38:00.775 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:38:00.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:38:00.776 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-154618822659 2020-03-11T22:38:00.776 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:38:00.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=154618822659 2020-03-11T22:38:00.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 154618822659' 2020-03-11T22:38:00.777 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:38:00.777 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 154618822659 2020-03-11T22:38:01.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 154618822659 -lt 154618822659 2020-03-11T22:38:01.119 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:38:01.119 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:01.119 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:01.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:38:01.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:38:01.565 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:38:01.565 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:38:01.565 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:38:01.565 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:38:01.565 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:38:01.565 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:38:01.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:38:01.905 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:38:01.905 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:01.905 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:02.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:38:02.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:38:02.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:38:02.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:38:02.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:38:02.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:38:02.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:38:02.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:38:02.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:38:02.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:38:02.461 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:38:02.461 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:38:02.461 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:38:02.461 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:38:02.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:38:02.786 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:38:02.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:02.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:03.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:38:03.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:38:03.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:38:03.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:38:03.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:38:03.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:38:03.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:38:03.226 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:38:03.226 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:38:03.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:38:03.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:38:03.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:38:03.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:38:03.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:38:03.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:38:03.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:38:03.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:38:03.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:38:03.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:38:03.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:38:03.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:38:03.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:38:04.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:38:04.225 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:38:04.226 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:04.226 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:04.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:38:04.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:38:04.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:38:04.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:38:04.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:38:04.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:38:04.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:38:04.667 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:38:04.667 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:38:05.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:38:05.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:38:05.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:38:05.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:38:05.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:38:05.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:38:05.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:38:05.517 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:38:05.517 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:38:05.518 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:38:05.518 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:38:05.518 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:38:05.518 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:38:05.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:38:05.849 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:38:05.849 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:05.849 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:06.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:38:06.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:38:06.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:38:06.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:38:06.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:38:06.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:38:06.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:38:06.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:38:06.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:38:06.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:38:06.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:38:06.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:38:06.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:38:06.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:38:07.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:38:07.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:38:07.567 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:38:07.567 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:38:07.567 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:38:07.567 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:38:07.568 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:38:07.568 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:38:07.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:38:07.898 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:38:07.898 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:07.898 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:08.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:38:08.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:38:08.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:38:08.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:38:08.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:38:08.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:38:08.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:38:08.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:38:08.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:38:08.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:38:08.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:38:08.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:38:08.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:38:08.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:38:10.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:38:10.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:38:10.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:38:10.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:38:10.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:38:10.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:38:10.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:38:10.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:38:10.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:38:10.729 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:38:10.730 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:10.730 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:11.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:38:11.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:38:11.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:38:11.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:38:11.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:38:11.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:38:11.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:38:11.170 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:38:11.170 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:38:11.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:38:11.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:38:11.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:38:11.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:38:11.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:38:14.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:38:14.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:38:14.835 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:38:14.835 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:38:14.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:38:14.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:38:14.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:38:14.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:38:15.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:38:15.175 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:38:15.176 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:15.176 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:15.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:38:15.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:38:15.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:38:15.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:38:15.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:38:15.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:38:15.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:38:15.638 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:38:15.639 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:38:16.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:38:16.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:38:16.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:38:16.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:38:16.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:38:22.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:38:22.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:38:22.478 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:38:22.478 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:38:22.478 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:38:22.478 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:38:22.479 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:38:22.479 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:38:22.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:38:22.819 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:38:22.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:22.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:23.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:38:23.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:38:23.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:38:23.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:38:23.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:38:23.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:38:23.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:38:23.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:38:23.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:38:23.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:38:23.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:38:23.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:38:23.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:38:23.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:38:31.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:38:31.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:38:31.007 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:38:31.007 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:38:31.007 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:38:31.008 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:38:31.008 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:38:31.008 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:38:31.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:38:31.347 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:38:31.347 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:31.347 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:31.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:38:31.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:38:31.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:38:31.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:38:31.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:38:31.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:38:31.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:38:31.792 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:38:31.793 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:38:32.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:38:32.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:38:32.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:38:32.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:38:32.688 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:38:32.688 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_fingerprint": "33fe8bbb-c194-472c-a065-7b61f5c4cae0", 2020-03-11T22:38:32.689 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:38:32.689 INFO:tasks.workunit.client.0.smithi145.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:38:32.689 INFO:tasks.workunit.client.0.smithi145.stdout: "timestamp": "2020-03-11T22:38:32.684534+0000", 2020-03-11T22:38:32.690 INFO:tasks.workunit.client.0.smithi145.stdout: "tag": "", 2020-03-11T22:38:32.690 INFO:tasks.workunit.client.0.smithi145.stdout: "health": { 2020-03-11T22:38:32.690 INFO:tasks.workunit.client.0.smithi145.stdout: "status": "HEALTH_WARN", 2020-03-11T22:38:32.690 INFO:tasks.workunit.client.0.smithi145.stdout: "checks": { 2020-03-11T22:38:32.691 INFO:tasks.workunit.client.0.smithi145.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:38:32.691 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:38:32.691 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2020-03-11T22:38:32.691 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:38:32.692 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 17 2020-03-11T22:38:32.692 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.692 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [], 2020-03-11T22:38:32.692 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2020-03-11T22:38:32.692 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.693 INFO:tasks.workunit.client.0.smithi145.stdout: "OSD_DOWN": { 2020-03-11T22:38:32.693 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:38:32.693 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2020-03-11T22:38:32.693 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "1 osds down", 2020-03-11T22:38:32.694 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 1 2020-03-11T22:38:32.694 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.694 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2020-03-11T22:38:32.694 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.695 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "osd.1 (root=default,host=smithi145) is down" 2020-03-11T22:38:32.695 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:38:32.695 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:38:32.695 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2020-03-11T22:38:32.696 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.696 INFO:tasks.workunit.client.0.smithi145.stdout: "PG_DEGRADED": { 2020-03-11T22:38:32.696 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:38:32.696 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2020-03-11T22:38:32.697 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2020-03-11T22:38:32.697 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 1 2020-03-11T22:38:32.697 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.697 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2020-03-11T22:38:32.698 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.698 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:38:32.698 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:38:32.698 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:38:32.699 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2020-03-11T22:38:32.699 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.699 INFO:tasks.workunit.client.0.smithi145.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:38:32.699 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:38:32.700 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2020-03-11T22:38:32.700 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:38:32.700 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 1 2020-03-11T22:38:32.700 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.700 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2020-03-11T22:38:32.701 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.701 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:38:32.701 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.701 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.702 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:38:32.702 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:38:32.702 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:38:32.702 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2020-03-11T22:38:32.702 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:38:32.703 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.703 INFO:tasks.workunit.client.0.smithi145.stdout: "mutes": [] 2020-03-11T22:38:32.703 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.703 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap_first_committed": 1, 2020-03-11T22:38:32.704 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap_last_committed": 1, 2020-03-11T22:38:32.704 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap": { 2020-03-11T22:38:32.704 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 1, 2020-03-11T22:38:32.704 INFO:tasks.workunit.client.0.smithi145.stdout: "fsid": "e665ac71-d8d3-412f-b706-8f1d72482f80", 2020-03-11T22:38:32.705 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2020-03-11T22:36:56.015610Z", 2020-03-11T22:38:32.705 INFO:tasks.workunit.client.0.smithi145.stdout: "created": "2020-03-11T22:36:56.015610Z", 2020-03-11T22:38:32.705 INFO:tasks.workunit.client.0.smithi145.stdout: "min_mon_release": 15, 2020-03-11T22:38:32.705 INFO:tasks.workunit.client.0.smithi145.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:38:32.706 INFO:tasks.workunit.client.0.smithi145.stdout: "features": { 2020-03-11T22:38:32.706 INFO:tasks.workunit.client.0.smithi145.stdout: "persistent": [ 2020-03-11T22:38:32.706 INFO:tasks.workunit.client.0.smithi145.stdout: "kraken", 2020-03-11T22:38:32.707 INFO:tasks.workunit.client.0.smithi145.stdout: "luminous", 2020-03-11T22:38:32.707 INFO:tasks.workunit.client.0.smithi145.stdout: "mimic", 2020-03-11T22:38:32.707 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap-prune", 2020-03-11T22:38:32.707 INFO:tasks.workunit.client.0.smithi145.stdout: "nautilus", 2020-03-11T22:38:32.708 INFO:tasks.workunit.client.0.smithi145.stdout: "octopus" 2020-03-11T22:38:32.708 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:38:32.708 INFO:tasks.workunit.client.0.smithi145.stdout: "optional": [] 2020-03-11T22:38:32.708 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.709 INFO:tasks.workunit.client.0.smithi145.stdout: "mons": [ 2020-03-11T22:38:32.709 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.709 INFO:tasks.workunit.client.0.smithi145.stdout: "rank": 0, 2020-03-11T22:38:32.709 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "a", 2020-03-11T22:38:32.710 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2020-03-11T22:38:32.710 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:38:32.710 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.710 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:38:32.711 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:38:32.711 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 0 2020-03-11T22:38:32.711 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:38:32.711 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:38:32.712 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.712 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:38:32.712 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:38:32.712 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 0, 2020-03-11T22:38:32.712 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 0 2020-03-11T22:38:32.713 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:38:32.713 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:38:32.713 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.713 INFO:tasks.workunit.client.0.smithi145.stdout: "quorum": [ 2020-03-11T22:38:32.714 INFO:tasks.workunit.client.0.smithi145.stdout: 0 2020-03-11T22:38:32.714 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:38:32.714 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap": { 2020-03-11T22:38:32.714 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 41, 2020-03-11T22:38:32.715 INFO:tasks.workunit.client.0.smithi145.stdout: "fsid": "e665ac71-d8d3-412f-b706-8f1d72482f80", 2020-03-11T22:38:32.715 INFO:tasks.workunit.client.0.smithi145.stdout: "created": "2020-03-11T22:36:56.779713+0000", 2020-03-11T22:38:32.715 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2020-03-11T22:37:57.990880+0000", 2020-03-11T22:38:32.715 INFO:tasks.workunit.client.0.smithi145.stdout: "last_up_change": "2020-03-11T22:37:56.234766+0000", 2020-03-11T22:38:32.715 INFO:tasks.workunit.client.0.smithi145.stdout: "last_in_change": "2020-03-11T22:37:57.720983+0000", 2020-03-11T22:38:32.716 INFO:tasks.workunit.client.0.smithi145.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:38:32.716 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_num": 5800384, 2020-03-11T22:38:32.716 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_set": [ 2020-03-11T22:38:32.716 INFO:tasks.workunit.client.0.smithi145.stdout: "nodown", 2020-03-11T22:38:32.717 INFO:tasks.workunit.client.0.smithi145.stdout: "noin", 2020-03-11T22:38:32.717 INFO:tasks.workunit.client.0.smithi145.stdout: "noout", 2020-03-11T22:38:32.717 INFO:tasks.workunit.client.0.smithi145.stdout: "pglog_hardlimit", 2020-03-11T22:38:32.718 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snapdirs", 2020-03-11T22:38:32.718 INFO:tasks.workunit.client.0.smithi145.stdout: "recovery_deletes", 2020-03-11T22:38:32.718 INFO:tasks.workunit.client.0.smithi145.stdout: "sortbitwise" 2020-03-11T22:38:32.718 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:38:32.719 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_version": 7, 2020-03-11T22:38:32.719 INFO:tasks.workunit.client.0.smithi145.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:38:32.719 INFO:tasks.workunit.client.0.smithi145.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:38:32.719 INFO:tasks.workunit.client.0.smithi145.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:38:32.720 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_snapshot": "", 2020-03-11T22:38:32.720 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_max": 1, 2020-03-11T22:38:32.720 INFO:tasks.workunit.client.0.smithi145.stdout: "max_osd": 3, 2020-03-11T22:38:32.720 INFO:tasks.workunit.client.0.smithi145.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:38:32.720 INFO:tasks.workunit.client.0.smithi145.stdout: "min_compat_client": "jewel", 2020-03-11T22:38:32.721 INFO:tasks.workunit.client.0.smithi145.stdout: "require_osd_release": "octopus", 2020-03-11T22:38:32.721 INFO:tasks.workunit.client.0.smithi145.stdout: "pools": [ 2020-03-11T22:38:32.721 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.721 INFO:tasks.workunit.client.0.smithi145.stdout: "pool": 1, 2020-03-11T22:38:32.722 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_name": "test", 2020-03-11T22:38:32.722 INFO:tasks.workunit.client.0.smithi145.stdout: "create_time": "2020-03-11T22:37:17.928906+0000", 2020-03-11T22:38:32.722 INFO:tasks.workunit.client.0.smithi145.stdout: "flags": 1, 2020-03-11T22:38:32.722 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_names": "hashpspool", 2020-03-11T22:38:32.723 INFO:tasks.workunit.client.0.smithi145.stdout: "type": 1, 2020-03-11T22:38:32.723 INFO:tasks.workunit.client.0.smithi145.stdout: "size": 3, 2020-03-11T22:38:32.723 INFO:tasks.workunit.client.0.smithi145.stdout: "min_size": 2, 2020-03-11T22:38:32.723 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_rule": 0, 2020-03-11T22:38:32.723 INFO:tasks.workunit.client.0.smithi145.stdout: "object_hash": 2, 2020-03-11T22:38:32.724 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_autoscale_mode": "off", 2020-03-11T22:38:32.724 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num": 1, 2020-03-11T22:38:32.724 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_placement_num": 1, 2020-03-11T22:38:32.724 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_placement_num_target": 1, 2020-03-11T22:38:32.725 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num_target": 1, 2020-03-11T22:38:32.725 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num_pending": 1, 2020-03-11T22:38:32.725 INFO:tasks.workunit.client.0.smithi145.stdout: "last_pg_merge_meta": { 2020-03-11T22:38:32.725 INFO:tasks.workunit.client.0.smithi145.stdout: "source_pgid": "0.0", 2020-03-11T22:38:32.726 INFO:tasks.workunit.client.0.smithi145.stdout: "ready_epoch": 0, 2020-03-11T22:38:32.726 INFO:tasks.workunit.client.0.smithi145.stdout: "last_epoch_started": 0, 2020-03-11T22:38:32.726 INFO:tasks.workunit.client.0.smithi145.stdout: "last_epoch_clean": 0, 2020-03-11T22:38:32.726 INFO:tasks.workunit.client.0.smithi145.stdout: "source_version": "0'0", 2020-03-11T22:38:32.726 INFO:tasks.workunit.client.0.smithi145.stdout: "target_version": "0'0" 2020-03-11T22:38:32.727 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.727 INFO:tasks.workunit.client.0.smithi145.stdout: "last_change": "29", 2020-03-11T22:38:32.727 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend": "0", 2020-03-11T22:38:32.727 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:38:32.728 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend_preluminous": "0", 2020-03-11T22:38:32.728 INFO:tasks.workunit.client.0.smithi145.stdout: "auid": 0, 2020-03-11T22:38:32.728 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_mode": "selfmanaged", 2020-03-11T22:38:32.728 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_seq": 0, 2020-03-11T22:38:32.729 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_epoch": 0, 2020-03-11T22:38:32.729 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_snaps": [], 2020-03-11T22:38:32.729 INFO:tasks.workunit.client.0.smithi145.stdout: "removed_snaps": "[]", 2020-03-11T22:38:32.729 INFO:tasks.workunit.client.0.smithi145.stdout: "quota_max_bytes": 0, 2020-03-11T22:38:32.729 INFO:tasks.workunit.client.0.smithi145.stdout: "quota_max_objects": 0, 2020-03-11T22:38:32.730 INFO:tasks.workunit.client.0.smithi145.stdout: "tiers": [], 2020-03-11T22:38:32.730 INFO:tasks.workunit.client.0.smithi145.stdout: "tier_of": -1, 2020-03-11T22:38:32.730 INFO:tasks.workunit.client.0.smithi145.stdout: "read_tier": -1, 2020-03-11T22:38:32.730 INFO:tasks.workunit.client.0.smithi145.stdout: "write_tier": -1, 2020-03-11T22:38:32.731 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_mode": "none", 2020-03-11T22:38:32.731 INFO:tasks.workunit.client.0.smithi145.stdout: "target_max_bytes": 0, 2020-03-11T22:38:32.731 INFO:tasks.workunit.client.0.smithi145.stdout: "target_max_objects": 0, 2020-03-11T22:38:32.731 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:38:32.732 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:38:32.732 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:38:32.732 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_min_flush_age": 0, 2020-03-11T22:38:32.732 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_min_evict_age": 0, 2020-03-11T22:38:32.733 INFO:tasks.workunit.client.0.smithi145.stdout: "erasure_code_profile": "", 2020-03-11T22:38:32.733 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_params": { 2020-03-11T22:38:32.733 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "none" 2020-03-11T22:38:32.733 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.733 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_period": 0, 2020-03-11T22:38:32.734 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_count": 0, 2020-03-11T22:38:32.734 INFO:tasks.workunit.client.0.smithi145.stdout: "use_gmt_hitset": true, 2020-03-11T22:38:32.734 INFO:tasks.workunit.client.0.smithi145.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:38:32.734 INFO:tasks.workunit.client.0.smithi145.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:38:32.735 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:38:32.735 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:38:32.735 INFO:tasks.workunit.client.0.smithi145.stdout: "grade_table": [], 2020-03-11T22:38:32.735 INFO:tasks.workunit.client.0.smithi145.stdout: "stripe_width": 0, 2020-03-11T22:38:32.735 INFO:tasks.workunit.client.0.smithi145.stdout: "expected_num_objects": 0, 2020-03-11T22:38:32.736 INFO:tasks.workunit.client.0.smithi145.stdout: "fast_read": false, 2020-03-11T22:38:32.736 INFO:tasks.workunit.client.0.smithi145.stdout: "options": {}, 2020-03-11T22:38:32.736 INFO:tasks.workunit.client.0.smithi145.stdout: "application_metadata": {} 2020-03-11T22:38:32.736 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:38:32.736 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:38:32.736 INFO:tasks.workunit.client.0.smithi145.stdout: "osds": [ 2020-03-11T22:38:32.736 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.736 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2020-03-11T22:38:32.737 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "85c0e5de-f314-4f60-acc1-ba357f36e17a", 2020-03-11T22:38:32.737 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 1, 2020-03-11T22:38:32.737 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2020-03-11T22:38:32.737 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2020-03-11T22:38:32.737 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2020-03-11T22:38:32.737 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 5, 2020-03-11T22:38:32.737 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 29, 2020-03-11T22:38:32.738 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 35, 2020-03-11T22:38:32.738 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 40, 2020-03-11T22:38:32.738 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 30, 2020-03-11T22:38:32.738 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2020-03-11T22:38:32.738 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2020-03-11T22:38:32.738 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:38:32.738 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.739 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:38:32.739 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:38:32.739 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 22517 2020-03-11T22:38:32.739 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.739 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.740 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:38:32.740 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:38:32.740 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 22517 2020-03-11T22:38:32.740 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:38:32.740 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:38:32.741 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.741 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2020-03-11T22:38:32.741 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:38:32.741 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.742 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:38:32.742 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:38:32.742 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 22517 2020-03-11T22:38:32.742 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.742 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.743 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:38:32.743 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:38:32.743 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 22517 2020-03-11T22:38:32.743 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:38:32.744 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:38:32.744 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.744 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2020-03-11T22:38:32.744 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:38:32.744 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.745 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:38:32.745 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:38:32.745 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 22517 2020-03-11T22:38:32.745 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.746 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.746 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:38:32.746 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:38:32.746 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 22517 2020-03-11T22:38:32.747 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:38:32.747 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:38:32.747 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.747 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2020-03-11T22:38:32.748 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:38:32.748 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.748 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:38:32.748 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:38:32.748 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 22517 2020-03-11T22:38:32.749 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.749 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.749 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:38:32.749 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:38:32.750 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 22517 2020-03-11T22:38:32.750 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:38:32.750 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:38:32.750 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.751 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6803/22517", 2020-03-11T22:38:32.751 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6805/22517", 2020-03-11T22:38:32.751 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6809/22517", 2020-03-11T22:38:32.751 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6807/22517", 2020-03-11T22:38:32.752 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2020-03-11T22:38:32.752 INFO:tasks.workunit.client.0.smithi145.stdout: "exists", 2020-03-11T22:38:32.752 INFO:tasks.workunit.client.0.smithi145.stdout: "up" 2020-03-11T22:38:32.752 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:38:32.753 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.753 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.753 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2020-03-11T22:38:32.753 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "29aaf178-2506-493e-b1ca-ca8045da325e", 2020-03-11T22:38:32.754 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 0, 2020-03-11T22:38:32.754 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2020-03-11T22:38:32.754 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2020-03-11T22:38:32.754 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2020-03-11T22:38:32.755 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 0, 2020-03-11T22:38:32.755 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 0, 2020-03-11T22:38:32.755 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 10, 2020-03-11T22:38:32.755 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 23, 2020-03-11T22:38:32.756 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 31, 2020-03-11T22:38:32.756 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2020-03-11T22:38:32.756 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2020-03-11T22:38:32.756 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:38:32.756 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.757 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:38:32.757 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:38:32.757 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 18194 2020-03-11T22:38:32.757 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.758 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.758 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:38:32.758 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:38:32.758 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 18194 2020-03-11T22:38:32.759 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:38:32.759 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:38:32.759 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.759 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2020-03-11T22:38:32.760 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:38:32.760 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.760 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:38:32.760 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:38:32.760 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 18194 2020-03-11T22:38:32.760 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.761 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.761 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:38:32.761 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:38:32.761 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 18194 2020-03-11T22:38:32.761 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:38:32.761 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:38:32.761 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.761 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2020-03-11T22:38:32.762 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:38:32.762 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.762 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:38:32.762 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:38:32.762 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 18194 2020-03-11T22:38:32.762 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.762 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.763 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:38:32.763 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:38:32.763 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 18194 2020-03-11T22:38:32.763 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:38:32.763 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:38:32.763 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.763 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2020-03-11T22:38:32.763 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:38:32.764 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.764 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:38:32.764 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:38:32.764 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 18194 2020-03-11T22:38:32.764 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.764 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.764 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:38:32.765 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:38:32.765 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 18194 2020-03-11T22:38:32.765 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:38:32.765 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:38:32.765 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.765 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6811/18194", 2020-03-11T22:38:32.765 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6813/18194", 2020-03-11T22:38:32.765 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6817/18194", 2020-03-11T22:38:32.766 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6815/18194", 2020-03-11T22:38:32.766 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2020-03-11T22:38:32.766 INFO:tasks.workunit.client.0.smithi145.stdout: "exists" 2020-03-11T22:38:32.766 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:38:32.766 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.766 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.766 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2020-03-11T22:38:32.767 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "52ae9eef-8cbf-42f9-836d-cf45b3fb4952", 2020-03-11T22:38:32.767 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 1, 2020-03-11T22:38:32.767 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2020-03-11T22:38:32.767 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2020-03-11T22:38:32.767 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2020-03-11T22:38:32.767 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 15, 2020-03-11T22:38:32.767 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 29, 2020-03-11T22:38:32.767 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 36, 2020-03-11T22:38:32.768 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 0, 2020-03-11T22:38:32.768 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 32, 2020-03-11T22:38:32.768 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2020-03-11T22:38:32.768 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2020-03-11T22:38:32.768 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:38:32.768 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.768 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:38:32.769 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:38:32.769 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 22711 2020-03-11T22:38:32.769 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.769 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.769 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:38:32.769 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:38:32.769 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 22711 2020-03-11T22:38:32.769 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:38:32.770 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:38:32.770 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.770 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2020-03-11T22:38:32.770 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:38:32.770 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.770 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:38:32.770 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:38:32.771 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 22711 2020-03-11T22:38:32.771 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.771 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.771 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:38:32.771 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:38:32.771 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 22711 2020-03-11T22:38:32.771 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:38:32.771 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:38:32.772 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.772 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2020-03-11T22:38:32.772 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:38:32.772 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.772 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:38:32.772 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:38:32.772 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 22711 2020-03-11T22:38:32.773 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.773 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.773 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:38:32.773 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:38:32.773 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 22711 2020-03-11T22:38:32.773 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:38:32.773 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:38:32.773 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.774 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2020-03-11T22:38:32.774 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:38:32.774 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.774 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:38:32.774 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:38:32.774 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 22711 2020-03-11T22:38:32.774 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.774 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.775 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:38:32.775 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:38:32.775 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 22711 2020-03-11T22:38:32.775 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:38:32.775 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:38:32.775 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.775 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6811/22711", 2020-03-11T22:38:32.776 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6813/22711", 2020-03-11T22:38:32.776 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6817/22711", 2020-03-11T22:38:32.776 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6815/22711", 2020-03-11T22:38:32.776 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2020-03-11T22:38:32.776 INFO:tasks.workunit.client.0.smithi145.stdout: "exists", 2020-03-11T22:38:32.776 INFO:tasks.workunit.client.0.smithi145.stdout: "up" 2020-03-11T22:38:32.776 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:38:32.776 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:38:32.777 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:38:32.777 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_xinfo": [ 2020-03-11T22:38:32.777 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.777 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2020-03-11T22:38:32.777 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "2020-03-11T22:37:50.105533+0000", 2020-03-11T22:38:32.777 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2020-03-11T22:38:32.777 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2020-03-11T22:38:32.778 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138292836696063, 2020-03-11T22:38:32.778 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2020-03-11T22:38:32.778 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2020-03-11T22:37:03.688736+0000", 2020-03-11T22:38:32.778 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2020-03-11T22:38:32.778 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.778 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.778 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2020-03-11T22:38:32.778 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "2020-03-11T22:37:50.593999+0000", 2020-03-11T22:38:32.779 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2020-03-11T22:38:32.779 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2020-03-11T22:38:32.779 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138292836696063, 2020-03-11T22:38:32.779 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2020-03-11T22:38:32.779 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2020-03-11T22:37:09.654001+0000", 2020-03-11T22:38:32.779 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2020-03-11T22:38:32.779 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.779 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.780 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2020-03-11T22:38:32.780 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "2020-03-11T22:37:51.085399+0000", 2020-03-11T22:38:32.780 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2020-03-11T22:38:32.780 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2020-03-11T22:38:32.780 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138292836696063, 2020-03-11T22:38:32.780 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2020-03-11T22:38:32.780 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2020-03-11T22:37:15.876315+0000", 2020-03-11T22:38:32.781 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2020-03-11T22:38:32.781 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:38:32.781 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:38:32.781 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_upmap": [], 2020-03-11T22:38:32.781 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_upmap_items": [], 2020-03-11T22:38:32.781 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_temp": [], 2020-03-11T22:38:32.781 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_temp": [], 2020-03-11T22:38:32.782 INFO:tasks.workunit.client.0.smithi145.stdout: "blacklist": {}, 2020-03-11T22:38:32.782 INFO:tasks.workunit.client.0.smithi145.stdout: "erasure_code_profiles": { 2020-03-11T22:38:32.782 INFO:tasks.workunit.client.0.smithi145.stdout: "default": { 2020-03-11T22:38:32.782 INFO:tasks.workunit.client.0.smithi145.stdout: "k": "2", 2020-03-11T22:38:32.782 INFO:tasks.workunit.client.0.smithi145.stdout: "m": "2", 2020-03-11T22:38:32.782 INFO:tasks.workunit.client.0.smithi145.stdout: "plugin": "jerasure", 2020-03-11T22:38:32.782 INFO:tasks.workunit.client.0.smithi145.stdout: "technique": "reed_sol_van" 2020-03-11T22:38:32.782 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:38:32.783 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.783 INFO:tasks.workunit.client.0.smithi145.stdout: "removed_snaps_queue": [], 2020-03-11T22:38:32.783 INFO:tasks.workunit.client.0.smithi145.stdout: "new_removed_snaps": [], 2020-03-11T22:38:32.783 INFO:tasks.workunit.client.0.smithi145.stdout: "new_purged_snaps": [], 2020-03-11T22:38:32.783 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_node_flags": {}, 2020-03-11T22:38:32.783 INFO:tasks.workunit.client.0.smithi145.stdout: "device_class_flags": {} 2020-03-11T22:38:32.783 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.784 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_metadata": [ 2020-03-11T22:38:32.784 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.784 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2020-03-11T22:38:32.784 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2020-03-11T22:38:32.784 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6804/22517,v1:127.0.0.1:6805/22517]", 2020-03-11T22:38:32.784 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2020-03-11T22:38:32.784 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2020-03-11T22:38:32.784 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:38:32.785 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:38:32.785 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:38:32.785 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:38:32.785 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:38:32.785 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:38:32.785 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:38:32.785 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:38:32.786 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:38:32.786 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:38:32.786 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:38:32.786 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:38:32.786 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "octopus", 2020-03-11T22:38:32.786 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:38:32.787 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:38:32.787 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:38:32.787 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2020-03-11T22:38:32.787 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XLDP", 2020-03-11T22:38:32.787 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:38:32.788 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2020-03-11T22:38:32.788 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2020-03-11T22:38:32.788 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:38:32.788 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2020-03-11T22:38:32.788 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6802/22517,v1:127.0.0.1:6803/22517]", 2020-03-11T22:38:32.788 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2020-03-11T22:38:32.789 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/22517,v1:127.0.0.1:6809/22517]", 2020-03-11T22:38:32.789 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/22517,v1:127.0.0.1:6807/22517]", 2020-03-11T22:38:32.789 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2020-03-11T22:38:32.789 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2020-03-11T22:38:32.789 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:38:32.789 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:38:32.789 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2020-03-11T22:38:32.790 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32655820", 2020-03-11T22:38:32.790 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:38:32.790 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:38:32.790 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2020-03-11T22:38:32.790 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:38:32.790 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:38:32.790 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2020-03-11T22:38:32.791 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.791 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.791 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2020-03-11T22:38:32.791 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2020-03-11T22:38:32.791 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6812/18194,v1:127.0.0.1:6813/18194]", 2020-03-11T22:38:32.791 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2020-03-11T22:38:32.791 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2020-03-11T22:38:32.792 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:38:32.792 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:38:32.792 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:38:32.792 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:38:32.792 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:38:32.792 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:38:32.792 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:38:32.792 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:38:32.793 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:38:32.793 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:38:32.793 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:38:32.793 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:38:32.793 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "octopus", 2020-03-11T22:38:32.793 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:38:32.793 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:38:32.794 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:38:32.794 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2020-03-11T22:38:32.794 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XLDP", 2020-03-11T22:38:32.794 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:38:32.794 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2020-03-11T22:38:32.794 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2020-03-11T22:38:32.794 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:38:32.795 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2020-03-11T22:38:32.795 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6810/18194,v1:127.0.0.1:6811/18194]", 2020-03-11T22:38:32.795 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2020-03-11T22:38:32.795 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/18194,v1:127.0.0.1:6817/18194]", 2020-03-11T22:38:32.795 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/18194,v1:127.0.0.1:6815/18194]", 2020-03-11T22:38:32.795 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2020-03-11T22:38:32.795 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2020-03-11T22:38:32.796 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:38:32.796 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:38:32.796 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2020-03-11T22:38:32.796 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32655820", 2020-03-11T22:38:32.796 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:38:32.796 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:38:32.796 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2020-03-11T22:38:32.796 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:38:32.797 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:38:32.797 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2020-03-11T22:38:32.797 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.797 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.797 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2020-03-11T22:38:32.798 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2020-03-11T22:38:32.798 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6812/22711,v1:127.0.0.1:6813/22711]", 2020-03-11T22:38:32.798 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2020-03-11T22:38:32.798 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2020-03-11T22:38:32.798 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:38:32.798 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:38:32.798 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:38:32.798 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:38:32.799 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:38:32.799 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:38:32.799 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:38:32.799 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:38:32.799 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:38:32.799 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:38:32.799 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:38:32.799 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:38:32.800 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "octopus", 2020-03-11T22:38:32.800 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:38:32.800 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:38:32.800 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:38:32.800 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2020-03-11T22:38:32.800 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XLDP", 2020-03-11T22:38:32.800 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:38:32.801 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2020-03-11T22:38:32.801 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2020-03-11T22:38:32.801 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:38:32.801 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2020-03-11T22:38:32.801 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6810/22711,v1:127.0.0.1:6811/22711]", 2020-03-11T22:38:32.801 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2020-03-11T22:38:32.801 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/22711,v1:127.0.0.1:6817/22711]", 2020-03-11T22:38:32.802 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/22711,v1:127.0.0.1:6815/22711]", 2020-03-11T22:38:32.802 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2020-03-11T22:38:32.802 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2020-03-11T22:38:32.802 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:38:32.802 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:38:32.802 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2020-03-11T22:38:32.802 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32655820", 2020-03-11T22:38:32.802 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:38:32.803 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:38:32.803 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2020-03-11T22:38:32.803 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:38:32.803 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:38:32.803 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2020-03-11T22:38:32.803 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:38:32.803 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:38:32.803 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap_first_committed": 1, 2020-03-11T22:38:32.804 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap_last_committed": 41, 2020-03-11T22:38:32.804 INFO:tasks.workunit.client.0.smithi145.stdout: "crushmap": { 2020-03-11T22:38:32.804 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": [ 2020-03-11T22:38:32.804 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.804 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2020-03-11T22:38:32.804 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.0", 2020-03-11T22:38:32.804 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2020-03-11T22:38:32.805 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.805 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.805 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2020-03-11T22:38:32.805 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.1", 2020-03-11T22:38:32.805 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2020-03-11T22:38:32.805 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.805 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.806 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2020-03-11T22:38:32.806 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.2", 2020-03-11T22:38:32.806 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2020-03-11T22:38:32.806 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:38:32.806 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:38:32.806 INFO:tasks.workunit.client.0.smithi145.stdout: "types": [ 2020-03-11T22:38:32.806 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.806 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 0, 2020-03-11T22:38:32.807 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd" 2020-03-11T22:38:32.807 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.807 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.807 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2020-03-11T22:38:32.807 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "host" 2020-03-11T22:38:32.807 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.808 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.808 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 2, 2020-03-11T22:38:32.808 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "chassis" 2020-03-11T22:38:32.808 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.808 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.808 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 3, 2020-03-11T22:38:32.808 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "rack" 2020-03-11T22:38:32.809 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.809 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.809 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 4, 2020-03-11T22:38:32.809 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "row" 2020-03-11T22:38:32.809 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.809 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.810 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 5, 2020-03-11T22:38:32.810 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "pdu" 2020-03-11T22:38:32.810 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.810 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.810 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 6, 2020-03-11T22:38:32.810 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "pod" 2020-03-11T22:38:32.810 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.810 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.811 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 7, 2020-03-11T22:38:32.811 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "room" 2020-03-11T22:38:32.811 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.811 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.811 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 8, 2020-03-11T22:38:32.811 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "datacenter" 2020-03-11T22:38:32.811 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.811 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.812 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 9, 2020-03-11T22:38:32.812 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "zone" 2020-03-11T22:38:32.812 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.812 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.812 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 10, 2020-03-11T22:38:32.812 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "region" 2020-03-11T22:38:32.812 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.813 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.813 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2020-03-11T22:38:32.813 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "root" 2020-03-11T22:38:32.813 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:38:32.813 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:38:32.813 INFO:tasks.workunit.client.0.smithi145.stdout: "buckets": [ 2020-03-11T22:38:32.813 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.813 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -1, 2020-03-11T22:38:32.814 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "default", 2020-03-11T22:38:32.814 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2020-03-11T22:38:32.814 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "root", 2020-03-11T22:38:32.814 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2020-03-11T22:38:32.814 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2020-03-11T22:38:32.814 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2020-03-11T22:38:32.814 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:38:32.815 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.815 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -3, 2020-03-11T22:38:32.815 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2020-03-11T22:38:32.815 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2020-03-11T22:38:32.815 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:38:32.815 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:38:32.815 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.815 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.816 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -2, 2020-03-11T22:38:32.816 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "default~hdd", 2020-03-11T22:38:32.816 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2020-03-11T22:38:32.816 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "root", 2020-03-11T22:38:32.816 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2020-03-11T22:38:32.816 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2020-03-11T22:38:32.816 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2020-03-11T22:38:32.817 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:38:32.817 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.817 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -4, 2020-03-11T22:38:32.817 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2020-03-11T22:38:32.817 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2020-03-11T22:38:32.817 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:38:32.817 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:38:32.818 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.818 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.818 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -3, 2020-03-11T22:38:32.818 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "smithi145", 2020-03-11T22:38:32.818 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2020-03-11T22:38:32.818 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "host", 2020-03-11T22:38:32.819 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2020-03-11T22:38:32.819 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2020-03-11T22:38:32.819 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2020-03-11T22:38:32.819 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:38:32.819 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.819 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2020-03-11T22:38:32.819 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2020-03-11T22:38:32.819 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2020-03-11T22:38:32.820 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.820 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.820 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2020-03-11T22:38:32.820 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2020-03-11T22:38:32.820 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 1 2020-03-11T22:38:32.820 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.820 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.820 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2020-03-11T22:38:32.821 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2020-03-11T22:38:32.821 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 2 2020-03-11T22:38:32.821 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:38:32.821 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:38:32.821 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.821 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.821 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -4, 2020-03-11T22:38:32.822 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "smithi145~hdd", 2020-03-11T22:38:32.822 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2020-03-11T22:38:32.822 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "host", 2020-03-11T22:38:32.822 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2020-03-11T22:38:32.822 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2020-03-11T22:38:32.822 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2020-03-11T22:38:32.822 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:38:32.822 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.823 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2020-03-11T22:38:32.823 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2020-03-11T22:38:32.823 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2020-03-11T22:38:32.823 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.823 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.823 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2020-03-11T22:38:32.823 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2020-03-11T22:38:32.824 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 1 2020-03-11T22:38:32.824 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.824 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.824 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2020-03-11T22:38:32.824 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2020-03-11T22:38:32.824 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 2 2020-03-11T22:38:32.824 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:38:32.825 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:38:32.825 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:38:32.825 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:38:32.825 INFO:tasks.workunit.client.0.smithi145.stdout: "rules": [ 2020-03-11T22:38:32.825 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.825 INFO:tasks.workunit.client.0.smithi145.stdout: "rule_id": 0, 2020-03-11T22:38:32.825 INFO:tasks.workunit.client.0.smithi145.stdout: "rule_name": "replicated_rule", 2020-03-11T22:38:32.825 INFO:tasks.workunit.client.0.smithi145.stdout: "ruleset": 0, 2020-03-11T22:38:32.826 INFO:tasks.workunit.client.0.smithi145.stdout: "type": 1, 2020-03-11T22:38:32.826 INFO:tasks.workunit.client.0.smithi145.stdout: "min_size": 1, 2020-03-11T22:38:32.826 INFO:tasks.workunit.client.0.smithi145.stdout: "max_size": 10, 2020-03-11T22:38:32.826 INFO:tasks.workunit.client.0.smithi145.stdout: "steps": [ 2020-03-11T22:38:32.826 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.826 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "take", 2020-03-11T22:38:32.826 INFO:tasks.workunit.client.0.smithi145.stdout: "item": -1, 2020-03-11T22:38:32.827 INFO:tasks.workunit.client.0.smithi145.stdout: "item_name": "default" 2020-03-11T22:38:32.827 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.827 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.827 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "choose_firstn", 2020-03-11T22:38:32.827 INFO:tasks.workunit.client.0.smithi145.stdout: "num": 0, 2020-03-11T22:38:32.827 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "osd" 2020-03-11T22:38:32.827 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.827 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.828 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "emit" 2020-03-11T22:38:32.828 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:38:32.828 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:38:32.828 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:38:32.828 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:38:32.828 INFO:tasks.workunit.client.0.smithi145.stdout: "tunables": { 2020-03-11T22:38:32.828 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_local_tries": 0, 2020-03-11T22:38:32.829 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:38:32.829 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_total_tries": 50, 2020-03-11T22:38:32.829 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:38:32.829 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:38:32.829 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_stable": 1, 2020-03-11T22:38:32.829 INFO:tasks.workunit.client.0.smithi145.stdout: "straw_calc_version": 1, 2020-03-11T22:38:32.829 INFO:tasks.workunit.client.0.smithi145.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:38:32.830 INFO:tasks.workunit.client.0.smithi145.stdout: "profile": "jewel", 2020-03-11T22:38:32.830 INFO:tasks.workunit.client.0.smithi145.stdout: "optimal_tunables": 1, 2020-03-11T22:38:32.830 INFO:tasks.workunit.client.0.smithi145.stdout: "legacy_tunables": 0, 2020-03-11T22:38:32.830 INFO:tasks.workunit.client.0.smithi145.stdout: "minimum_required_version": "jewel", 2020-03-11T22:38:32.830 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables": 1, 2020-03-11T22:38:32.830 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables2": 1, 2020-03-11T22:38:32.830 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v2_rules": 0, 2020-03-11T22:38:32.830 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables3": 1, 2020-03-11T22:38:32.831 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v3_rules": 0, 2020-03-11T22:38:32.831 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v4_buckets": 1, 2020-03-11T22:38:32.831 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables5": 1, 2020-03-11T22:38:32.831 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v5_rules": 0 2020-03-11T22:38:32.831 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.831 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_args": {} 2020-03-11T22:38:32.831 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.831 INFO:tasks.workunit.client.0.smithi145.stdout: "fsmap": { 2020-03-11T22:38:32.832 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 1, 2020-03-11T22:38:32.832 INFO:tasks.workunit.client.0.smithi145.stdout: "default_fscid": -1, 2020-03-11T22:38:32.832 INFO:tasks.workunit.client.0.smithi145.stdout: "compat": { 2020-03-11T22:38:32.832 INFO:tasks.workunit.client.0.smithi145.stdout: "compat": {}, 2020-03-11T22:38:32.832 INFO:tasks.workunit.client.0.smithi145.stdout: "ro_compat": {}, 2020-03-11T22:38:32.832 INFO:tasks.workunit.client.0.smithi145.stdout: "incompat": { 2020-03-11T22:38:32.832 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_1": "base v0.20", 2020-03-11T22:38:32.833 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:38:32.833 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:38:32.833 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:38:32.833 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:38:32.833 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:38:32.833 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_8": "no anchor table", 2020-03-11T22:38:32.833 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_9": "file layout v2", 2020-03-11T22:38:32.833 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:38:32.834 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:38:32.834 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.834 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_flags": { 2020-03-11T22:38:32.834 INFO:tasks.workunit.client.0.smithi145.stdout: "enable_multiple": false, 2020-03-11T22:38:32.834 INFO:tasks.workunit.client.0.smithi145.stdout: "ever_enabled_multiple": false 2020-03-11T22:38:32.834 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.834 INFO:tasks.workunit.client.0.smithi145.stdout: "standbys": [], 2020-03-11T22:38:32.835 INFO:tasks.workunit.client.0.smithi145.stdout: "filesystems": [] 2020-03-11T22:38:32.835 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.835 INFO:tasks.workunit.client.0.smithi145.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:38:32.835 INFO:tasks.workunit.client.0.smithi145.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:38:32.835 INFO:tasks.workunit.client.0.smithi145.stdout: "auth": { 2020-03-11T22:38:32.835 INFO:tasks.workunit.client.0.smithi145.stdout: "first_committed": 1, 2020-03-11T22:38:32.835 INFO:tasks.workunit.client.0.smithi145.stdout: "last_committed": 5, 2020-03-11T22:38:32.835 INFO:tasks.workunit.client.0.smithi145.stdout: "num_secrets": 3 2020-03-11T22:38:32.836 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.836 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg": 1, 2020-03-11T22:38:32.836 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_active": 1, 2020-03-11T22:38:32.836 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_unknown": 0, 2020-03-11T22:38:32.836 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osd": 3, 2020-03-11T22:38:32.836 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_sum": { 2020-03-11T22:38:32.836 INFO:tasks.workunit.client.0.smithi145.stdout: "stat_sum": { 2020-03-11T22:38:32.837 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes": 48300, 2020-03-11T22:38:32.837 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects": 100, 2020-03-11T22:38:32.837 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_clones": 0, 2020-03-11T22:38:32.837 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_copies": 300, 2020-03-11T22:38:32.837 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:38:32.837 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing": 0, 2020-03-11T22:38:32.837 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_degraded": 100, 2020-03-11T22:38:32.837 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_misplaced": 0, 2020-03-11T22:38:32.838 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_unfound": 0, 2020-03-11T22:38:32.838 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_dirty": 100, 2020-03-11T22:38:32.838 INFO:tasks.workunit.client.0.smithi145.stdout: "num_whiteouts": 0, 2020-03-11T22:38:32.838 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read": 0, 2020-03-11T22:38:32.838 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read_kb": 0, 2020-03-11T22:38:32.838 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write": 100, 2020-03-11T22:38:32.838 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write_kb": 100, 2020-03-11T22:38:32.839 INFO:tasks.workunit.client.0.smithi145.stdout: "num_scrub_errors": 0, 2020-03-11T22:38:32.839 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:38:32.839 INFO:tasks.workunit.client.0.smithi145.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:38:32.839 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_recovered": 0, 2020-03-11T22:38:32.839 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_recovered": 0, 2020-03-11T22:38:32.839 INFO:tasks.workunit.client.0.smithi145.stdout: "num_keys_recovered": 0, 2020-03-11T22:38:32.839 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_omap": 0, 2020-03-11T22:38:32.839 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:38:32.840 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:38:32.840 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush": 0, 2020-03-11T22:38:32.840 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_kb": 0, 2020-03-11T22:38:32.840 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict": 0, 2020-03-11T22:38:32.840 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_kb": 0, 2020-03-11T22:38:32.840 INFO:tasks.workunit.client.0.smithi145.stdout: "num_promote": 0, 2020-03-11T22:38:32.840 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_high": 0, 2020-03-11T22:38:32.840 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_low": 0, 2020-03-11T22:38:32.841 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_some": 0, 2020-03-11T22:38:32.841 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_full": 0, 2020-03-11T22:38:32.841 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_pinned": 0, 2020-03-11T22:38:32.841 INFO:tasks.workunit.client.0.smithi145.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:38:32.841 INFO:tasks.workunit.client.0.smithi145.stdout: "num_large_omap_objects": 0, 2020-03-11T22:38:32.841 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_manifest": 0, 2020-03-11T22:38:32.841 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_bytes": 0, 2020-03-11T22:38:32.841 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_keys": 0, 2020-03-11T22:38:32.842 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_repaired": 0 2020-03-11T22:38:32.842 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.842 INFO:tasks.workunit.client.0.smithi145.stdout: "store_stats": { 2020-03-11T22:38:32.842 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 0, 2020-03-11T22:38:32.842 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 0, 2020-03-11T22:38:32.842 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2020-03-11T22:38:32.842 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 0, 2020-03-11T22:38:32.843 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 0, 2020-03-11T22:38:32.843 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2020-03-11T22:38:32.843 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2020-03-11T22:38:32.843 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2020-03-11T22:38:32.843 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 0, 2020-03-11T22:38:32.843 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 0 2020-03-11T22:38:32.843 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.843 INFO:tasks.workunit.client.0.smithi145.stdout: "log_size": 16, 2020-03-11T22:38:32.844 INFO:tasks.workunit.client.0.smithi145.stdout: "ondisk_log_size": 16, 2020-03-11T22:38:32.844 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 2, 2020-03-11T22:38:32.844 INFO:tasks.workunit.client.0.smithi145.stdout: "acting": 2, 2020-03-11T22:38:32.844 INFO:tasks.workunit.client.0.smithi145.stdout: "num_store_stats": 0 2020-03-11T22:38:32.844 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.844 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_sum": { 2020-03-11T22:38:32.844 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 0, 2020-03-11T22:38:32.844 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 0, 2020-03-11T22:38:32.845 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pgs": 3, 2020-03-11T22:38:32.845 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osds": 3, 2020-03-11T22:38:32.845 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_osds": 3, 2020-03-11T22:38:32.845 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:38:32.845 INFO:tasks.workunit.client.0.smithi145.stdout: "kb": 314572800, 2020-03-11T22:38:32.845 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used": 3179776, 2020-03-11T22:38:32.845 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_data": 33792, 2020-03-11T22:38:32.845 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_omap": 13, 2020-03-11T22:38:32.846 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_meta": 3145714, 2020-03-11T22:38:32.846 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_avail": 311393024, 2020-03-11T22:38:32.846 INFO:tasks.workunit.client.0.smithi145.stdout: "statfs": { 2020-03-11T22:38:32.846 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 322122547200, 2020-03-11T22:38:32.846 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 318866456576, 2020-03-11T22:38:32.847 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 3221225472, 2020-03-11T22:38:32.847 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 34603008, 2020-03-11T22:38:32.847 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 416297, 2020-03-11T22:38:32.847 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2020-03-11T22:38:32.847 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2020-03-11T22:38:32.847 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2020-03-11T22:38:32.847 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 13438, 2020-03-11T22:38:32.847 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 3221212034 2020-03-11T22:38:32.848 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.848 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_peers": [], 2020-03-11T22:38:32.848 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:38:32.848 INFO:tasks.workunit.client.0.smithi145.stdout: "num_snap_trimming": 0, 2020-03-11T22:38:32.848 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shards_repaired": 1, 2020-03-11T22:38:32.848 INFO:tasks.workunit.client.0.smithi145.stdout: "op_queue_age_hist": { 2020-03-11T22:38:32.848 INFO:tasks.workunit.client.0.smithi145.stdout: "histogram": [], 2020-03-11T22:38:32.849 INFO:tasks.workunit.client.0.smithi145.stdout: "upper_bound": 1 2020-03-11T22:38:32.849 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.849 INFO:tasks.workunit.client.0.smithi145.stdout: "perf_stat": { 2020-03-11T22:38:32.849 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ms": 0, 2020-03-11T22:38:32.849 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ms": 0, 2020-03-11T22:38:32.849 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ns": 0, 2020-03-11T22:38:32.849 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ns": 0 2020-03-11T22:38:32.849 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.850 INFO:tasks.workunit.client.0.smithi145.stdout: "alerts": [], 2020-03-11T22:38:32.850 INFO:tasks.workunit.client.0.smithi145.stdout: "network_ping_times": [] 2020-03-11T22:38:32.850 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.850 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_sum_by_class": { 2020-03-11T22:38:32.850 INFO:tasks.workunit.client.0.smithi145.stdout: "hdd": { 2020-03-11T22:38:32.850 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 0, 2020-03-11T22:38:32.850 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 0, 2020-03-11T22:38:32.850 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pgs": 3, 2020-03-11T22:38:32.851 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osds": 3, 2020-03-11T22:38:32.851 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_osds": 3, 2020-03-11T22:38:32.851 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:38:32.851 INFO:tasks.workunit.client.0.smithi145.stdout: "kb": 314572800, 2020-03-11T22:38:32.851 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used": 3179776, 2020-03-11T22:38:32.851 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_data": 33792, 2020-03-11T22:38:32.851 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_omap": 13, 2020-03-11T22:38:32.852 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_meta": 3145714, 2020-03-11T22:38:32.852 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_avail": 311393024, 2020-03-11T22:38:32.852 INFO:tasks.workunit.client.0.smithi145.stdout: "statfs": { 2020-03-11T22:38:32.852 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 322122547200, 2020-03-11T22:38:32.852 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 318866456576, 2020-03-11T22:38:32.852 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 3221225472, 2020-03-11T22:38:32.852 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 34603008, 2020-03-11T22:38:32.852 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 416297, 2020-03-11T22:38:32.853 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2020-03-11T22:38:32.853 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2020-03-11T22:38:32.853 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2020-03-11T22:38:32.853 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 13438, 2020-03-11T22:38:32.853 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 3221212034 2020-03-11T22:38:32.853 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.854 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_peers": [], 2020-03-11T22:38:32.854 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:38:32.854 INFO:tasks.workunit.client.0.smithi145.stdout: "num_snap_trimming": 0, 2020-03-11T22:38:32.854 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shards_repaired": 1, 2020-03-11T22:38:32.854 INFO:tasks.workunit.client.0.smithi145.stdout: "op_queue_age_hist": { 2020-03-11T22:38:32.854 INFO:tasks.workunit.client.0.smithi145.stdout: "histogram": [], 2020-03-11T22:38:32.854 INFO:tasks.workunit.client.0.smithi145.stdout: "upper_bound": 1 2020-03-11T22:38:32.855 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.855 INFO:tasks.workunit.client.0.smithi145.stdout: "perf_stat": { 2020-03-11T22:38:32.855 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ms": 0, 2020-03-11T22:38:32.855 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ms": 0, 2020-03-11T22:38:32.855 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ns": 0, 2020-03-11T22:38:32.855 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ns": 0 2020-03-11T22:38:32.855 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.855 INFO:tasks.workunit.client.0.smithi145.stdout: "alerts": [], 2020-03-11T22:38:32.856 INFO:tasks.workunit.client.0.smithi145.stdout: "network_ping_times": [] 2020-03-11T22:38:32.856 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:38:32.856 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.856 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_stats": [ 2020-03-11T22:38:32.856 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.856 INFO:tasks.workunit.client.0.smithi145.stdout: "poolid": 1, 2020-03-11T22:38:32.856 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg": 1, 2020-03-11T22:38:32.856 INFO:tasks.workunit.client.0.smithi145.stdout: "stat_sum": { 2020-03-11T22:38:32.857 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes": 48300, 2020-03-11T22:38:32.857 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects": 100, 2020-03-11T22:38:32.857 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_clones": 0, 2020-03-11T22:38:32.857 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_copies": 300, 2020-03-11T22:38:32.857 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:38:32.857 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing": 0, 2020-03-11T22:38:32.857 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_degraded": 100, 2020-03-11T22:38:32.858 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_misplaced": 0, 2020-03-11T22:38:32.858 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_unfound": 0, 2020-03-11T22:38:32.858 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_dirty": 100, 2020-03-11T22:38:32.858 INFO:tasks.workunit.client.0.smithi145.stdout: "num_whiteouts": 0, 2020-03-11T22:38:32.858 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read": 0, 2020-03-11T22:38:32.858 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read_kb": 0, 2020-03-11T22:38:32.858 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write": 100, 2020-03-11T22:38:32.858 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write_kb": 100, 2020-03-11T22:38:32.859 INFO:tasks.workunit.client.0.smithi145.stdout: "num_scrub_errors": 0, 2020-03-11T22:38:32.859 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:38:32.859 INFO:tasks.workunit.client.0.smithi145.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:38:32.859 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_recovered": 0, 2020-03-11T22:38:32.859 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_recovered": 0, 2020-03-11T22:38:32.859 INFO:tasks.workunit.client.0.smithi145.stdout: "num_keys_recovered": 0, 2020-03-11T22:38:32.859 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_omap": 0, 2020-03-11T22:38:32.860 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:38:32.860 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:38:32.860 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush": 0, 2020-03-11T22:38:32.861 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_kb": 0, 2020-03-11T22:38:32.861 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict": 0, 2020-03-11T22:38:32.861 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_kb": 0, 2020-03-11T22:38:32.861 INFO:tasks.workunit.client.0.smithi145.stdout: "num_promote": 0, 2020-03-11T22:38:32.861 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_high": 0, 2020-03-11T22:38:32.861 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_low": 0, 2020-03-11T22:38:32.862 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_some": 0, 2020-03-11T22:38:32.862 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_full": 0, 2020-03-11T22:38:32.862 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_pinned": 0, 2020-03-11T22:38:32.862 INFO:tasks.workunit.client.0.smithi145.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:38:32.862 INFO:tasks.workunit.client.0.smithi145.stdout: "num_large_omap_objects": 0, 2020-03-11T22:38:32.862 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_manifest": 0, 2020-03-11T22:38:32.863 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_bytes": 0, 2020-03-11T22:38:32.863 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_keys": 0, 2020-03-11T22:38:32.863 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_repaired": 0 2020-03-11T22:38:32.863 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.863 INFO:tasks.workunit.client.0.smithi145.stdout: "store_stats": { 2020-03-11T22:38:32.863 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 0, 2020-03-11T22:38:32.863 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 0, 2020-03-11T22:38:32.864 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2020-03-11T22:38:32.864 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 19660800, 2020-03-11T22:38:32.864 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 144900, 2020-03-11T22:38:32.864 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2020-03-11T22:38:32.864 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2020-03-11T22:38:32.864 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2020-03-11T22:38:32.864 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 0, 2020-03-11T22:38:32.864 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 0 2020-03-11T22:38:32.865 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.865 INFO:tasks.workunit.client.0.smithi145.stdout: "log_size": 16, 2020-03-11T22:38:32.865 INFO:tasks.workunit.client.0.smithi145.stdout: "ondisk_log_size": 16, 2020-03-11T22:38:32.865 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 2, 2020-03-11T22:38:32.865 INFO:tasks.workunit.client.0.smithi145.stdout: "acting": 2, 2020-03-11T22:38:32.865 INFO:tasks.workunit.client.0.smithi145.stdout: "num_store_stats": 3 2020-03-11T22:38:32.865 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:38:32.865 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:38:32.866 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_stats": [ 2020-03-11T22:38:32.866 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.866 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2020-03-11T22:38:32.866 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 150323855370 2020-03-11T22:38:32.866 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.866 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.866 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2020-03-11T22:38:32.867 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 42949672975 2020-03-11T22:38:32.867 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.867 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.867 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2020-03-11T22:38:32.867 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 154618822666 2020-03-11T22:38:32.867 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:38:32.867 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:38:32.867 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_by_state": [ 2020-03-11T22:38:32.868 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.868 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "active+undersized+degraded", 2020-03-11T22:38:32.868 INFO:tasks.workunit.client.0.smithi145.stdout: "num": 1 2020-03-11T22:38:32.868 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:38:32.868 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:38:32.868 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_by_osd": [ 2020-03-11T22:38:32.868 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.869 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2020-03-11T22:38:32.869 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 0, 2020-03-11T22:38:32.869 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 1, 2020-03-11T22:38:32.869 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:38:32.869 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.869 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.869 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2020-03-11T22:38:32.869 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 0, 2020-03-11T22:38:32.870 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 0, 2020-03-11T22:38:32.870 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:38:32.870 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.870 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.870 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2020-03-11T22:38:32.870 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 1, 2020-03-11T22:38:32.870 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 1, 2020-03-11T22:38:32.871 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:38:32.871 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:38:32.871 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:38:32.871 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snaps": [ 2020-03-11T22:38:32.871 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:38:32.871 INFO:tasks.workunit.client.0.smithi145.stdout: "pool": 1, 2020-03-11T22:38:32.871 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snaps": {} 2020-03-11T22:38:32.871 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:38:32.872 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:38:32.872 INFO:tasks.workunit.client.0.smithi145.stdout: "servicemap": { 2020-03-11T22:38:32.872 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 1, 2020-03-11T22:38:32.872 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2020-03-11T22:36:56.780017+0000", 2020-03-11T22:38:32.872 INFO:tasks.workunit.client.0.smithi145.stdout: "services": {} 2020-03-11T22:38:32.872 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:38:32.872 INFO:tasks.workunit.client.0.smithi145.stdout: "paxos": { 2020-03-11T22:38:32.872 INFO:tasks.workunit.client.0.smithi145.stdout: "first_committed": 1, 2020-03-11T22:38:32.873 INFO:tasks.workunit.client.0.smithi145.stdout: "last_committed": 303, 2020-03-11T22:38:32.873 INFO:tasks.workunit.client.0.smithi145.stdout: "last_pn": 0, 2020-03-11T22:38:32.873 INFO:tasks.workunit.client.0.smithi145.stdout: "accepted_pn": 0 2020-03-11T22:38:32.873 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:38:32.873 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:38:32.874 INFO:tasks.workunit.client.0.smithi145.stderr:report 2856133943 2020-03-11T22:38:32.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:38:32.875 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: expr 2 + 2 2020-03-11T22:38:32.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: objname=existing_4 2020-03-11T22:38:32.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:154: TEST_divergent: echo 'writing non-divergent object existing_4' 2020-03-11T22:38:32.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:155: TEST_divergent: ceph pg dump pgs 2020-03-11T22:38:32.875 INFO:tasks.workunit.client.0.smithi145.stdout:writing non-divergent object existing_4 2020-03-11T22:38:33.018 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:33.019 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2020-03-11T22:38:13.669514+0000 29'100 41:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:37:17.979060+0000 0'0 2020-03-11T22:37:17.979060+0000 0 2020-03-11T22:38:33.019 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:38:33.019 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:33.020 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:38:33.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:156: TEST_divergent: rados -p test put existing_4 /etc/resolv.conf 2020-03-11T22:38:33.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:159: TEST_divergent: echo 'delay recovery' 2020-03-11T22:38:33.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:160: TEST_divergent: ceph pg dump pgs 2020-03-11T22:38:33.065 INFO:tasks.workunit.client.0.smithi145.stdout:delay recovery 2020-03-11T22:38:33.384 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:33.384 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2020-03-11T22:38:13.669514+0000 29'100 41:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:37:17.979060+0000 0'0 2020-03-11T22:37:17.979060+0000 0 2020-03-11T22:38:33.384 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:38:33.384 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:33.385 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:38:33.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2020-03-11T22:38:33.394 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.0 2020-03-11T22:38:33.395 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:38:33.395 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:38:33.395 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:38:33.395 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:33.395 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:38:33.395 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17402/ceph-osd.0.asok 2020-03-11T22:38:33.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2020-03-11T22:38:33.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.17402/ceph-osd.0.asok set_recovery_delay 100000 2020-03-11T22:38:33.486 INFO:tasks.workunit.client.0.smithi145.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:38:33.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2020-03-11T22:38:33.494 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.2 2020-03-11T22:38:33.494 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:38:33.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:38:33.495 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:38:33.495 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:33.495 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:38:33.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17402/ceph-osd.2.asok 2020-03-11T22:38:33.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2020-03-11T22:38:33.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.17402/ceph-osd.2.asok set_recovery_delay 100000 2020-03-11T22:38:33.585 INFO:tasks.workunit.client.0.smithi145.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:38:33.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:167: TEST_divergent: echo 'revive divergent 1' 2020-03-11T22:38:33.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:168: TEST_divergent: ceph pg dump pgs 2020-03-11T22:38:33.593 INFO:tasks.workunit.client.0.smithi145.stdout:revive divergent 1 2020-03-11T22:38:33.912 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:33.912 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2020-03-11T22:38:13.669514+0000 29'100 41:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:37:17.979060+0000 0'0 2020-03-11T22:37:17.979060+0000 0 2020-03-11T22:38:33.912 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:38:33.912 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:33.913 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:38:33.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:169: TEST_divergent: ceph osd set noup 2020-03-11T22:38:34.534 INFO:tasks.workunit.client.0.smithi145.stderr:noup is set 2020-03-11T22:38:34.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:170: TEST_divergent: activate_osd td/divergent-priors 1 2020-03-11T22:38:34.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:38:34.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:38:34.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:38:34.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:38:34.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:38:34.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:38:34.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:38:34.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:38:34.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:38:34.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:38:34.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:38:34.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:38:34.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:38:34.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:38:34.552 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:38:34.552 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:38:34.552 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:38:34.552 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:38:34.552 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:34.553 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:38:34.553 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:38:34.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:38:34.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:38:34.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:38:34.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:38:34.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:38:34.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:38:34.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:38:34.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:38:34.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:38:34.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:38:34.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:38:34.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:38:34.555 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:38:34.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:38:34.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:38:34.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:38:34.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:38:34.555 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2020-03-11T22:38:34.572 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:38:34.569+0000 7fcbae7b1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:34.573 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:38:34.570+0000 7fcbae7b1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:34.574 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:38:34.572+0000 7fcbae7b1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:34.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: return 0 2020-03-11T22:38:34.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:171: TEST_divergent: sleep 5 2020-03-11T22:38:35.090 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:38:35.087+0000 7fcbae7b1ec0 -1 Falling back to public interface 2020-03-11T22:38:35.922 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:38:35.920+0000 7fcbae7b1ec0 -1 osd.1 29 log_to_monitors {default=true} 2020-03-11T22:38:39.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:173: TEST_divergent: echo 'delay recovery divergent' 2020-03-11T22:38:39.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:174: TEST_divergent: ceph pg dump pgs 2020-03-11T22:38:39.895 INFO:tasks.workunit.client.0.smithi145.stdout:delay recovery divergent 2020-03-11T22:38:40.223 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:40.223 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:38:13.669514+0000 41'101 42:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:37:17.979060+0000 0'0 2020-03-11T22:37:17.979060+0000 0 2020-03-11T22:38:40.223 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:38:40.223 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:40.224 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:38:40.233 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: get_asok_path osd.1 2020-03-11T22:38:40.234 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:38:40.234 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:38:40.234 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:38:40.234 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:40.234 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:38:40.234 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17402/ceph-osd.1.asok 2020-03-11T22:38:40.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: CEPH_ARGS= 2020-03-11T22:38:40.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.17402/ceph-osd.1.asok set_recovery_delay 100000 2020-03-11T22:38:40.325 INFO:tasks.workunit.client.0.smithi145.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:38:40.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:177: TEST_divergent: ceph osd unset noup 2020-03-11T22:38:41.023 INFO:tasks.workunit.client.0.smithi145.stderr:noup is unset 2020-03-11T22:38:41.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:179: TEST_divergent: wait_for_osd up 0 2020-03-11T22:38:41.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:38:41.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:38:41.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:38:41.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:38:41.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:38:41.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:38:41.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:38:41.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:38:41.033 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:38:41.379 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 35 up_thru 40 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/22517,v1:127.0.0.1:6803/22517] [v2:127.0.0.1:6804/22517,v1:127.0.0.1:6805/22517] exists,up 85c0e5de-f314-4f60-acc1-ba357f36e17a 2020-03-11T22:38:41.379 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:38:41.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:38:41.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:38:41.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:38:41.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:180: TEST_divergent: wait_for_osd up 1 2020-03-11T22:38:41.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:38:41.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:38:41.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:38:41.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:38:41.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:38:41.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:38:41.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:38:41.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:38:41.720 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 45 up_thru 45 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/24171,v1:127.0.0.1:6819/24171] [v2:127.0.0.1:6820/24171,v1:127.0.0.1:6821/24171] exists,up 29aaf178-2506-493e-b1ca-ca8045da325e 2020-03-11T22:38:41.721 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:38:41.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:38:41.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:38:41.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:38:41.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:181: TEST_divergent: wait_for_osd up 2 2020-03-11T22:38:41.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:38:41.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:38:41.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:38:41.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:38:41.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:38:41.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:38:41.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:38:41.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:38:42.068 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 36 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/22711,v1:127.0.0.1:6811/22711] [v2:127.0.0.1:6812/22711,v1:127.0.0.1:6813/22711] exists,up 52ae9eef-8cbf-42f9-836d-cf45b3fb4952 2020-03-11T22:38:42.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:38:42.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:38:42.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:38:42.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:183: TEST_divergent: ceph pg dump pgs 2020-03-11T22:38:42.427 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:42.428 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 0 0 0 48300 0 0 17 17 peering 2020-03-11T22:38:40.877695+0000 29'101 45:124 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:37:17.979060+0000 0'0 2020-03-11T22:37:17.979060+0000 0 2020-03-11T22:38:42.428 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:38:42.428 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:42.429 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:38:42.437 INFO:tasks.workunit.client.0.smithi145.stdout:wait for peering 2020-03-11T22:38:42.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:184: TEST_divergent: echo 'wait for peering' 2020-03-11T22:38:42.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:185: TEST_divergent: ceph pg dump pgs 2020-03-11T22:38:42.755 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:42.755 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 0 0 0 48300 0 0 17 17 peering 2020-03-11T22:38:40.877695+0000 29'101 45:124 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:37:17.979060+0000 0'0 2020-03-11T22:37:17.979060+0000 0 2020-03-11T22:38:42.756 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:38:42.756 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:42.757 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:38:42.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:186: TEST_divergent: rados -p test put foo /etc/fstab 2020-03-11T22:38:54.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:188: TEST_divergent: echo 'killing divergent 1' 2020-03-11T22:38:54.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:189: TEST_divergent: ceph pg dump pgs 2020-03-11T22:38:54.155 INFO:tasks.workunit.client.0.smithi145.stdout:killing divergent 1 2020-03-11T22:38:54.482 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:54.523 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 2 2 0 0 47947 0 0 17 17 active+recovering+degraded+wait 2020-03-11T22:38:41.249249+0000 41'101 46:133 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:37:17.979060+0000 0'0 2020-03-11T22:37:17.979060+0000 0 2020-03-11T22:38:54.524 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:38:54.524 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:54.525 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:38:54.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:190: TEST_divergent: kill_daemons td/divergent-priors KILL osd.1 2020-03-11T22:38:54.525 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:38:54.525 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:38:54.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:38:54.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:38:54.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:38:54.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:38:54.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:192: TEST_divergent: echo 'reviving divergent 1' 2020-03-11T22:38:54.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:193: TEST_divergent: ceph pg dump pgs 2020-03-11T22:38:54.599 INFO:tasks.workunit.client.0.smithi145.stdout:reviving divergent 1 2020-03-11T22:38:54.918 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:54.918 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 2 2 0 0 47947 0 0 17 17 active+recovering+degraded+wait 2020-03-11T22:38:41.249249+0000 41'101 46:133 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:37:17.979060+0000 0'0 2020-03-11T22:37:17.979060+0000 0 2020-03-11T22:38:54.918 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:38:54.919 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:54.920 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:38:54.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:194: TEST_divergent: activate_osd td/divergent-priors 1 2020-03-11T22:38:54.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:38:54.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:38:54.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:38:54.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:38:54.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:38:54.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:38:54.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:38:54.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:38:54.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:38:54.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:38:54.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:38:54.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:38:54.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:38:54.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:38:54.933 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:38:54.934 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:38:54.934 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:38:54.934 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:38:54.934 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:54.935 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:38:54.935 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:38:54.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:38:54.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:38:54.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:38:54.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:38:54.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:38:54.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:38:54.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:38:54.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:38:54.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:38:54.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:38:54.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:38:54.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:38:54.937 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:38:54.937 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2020-03-11T22:38:54.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:38:54.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:38:54.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:38:54.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:38:54.957 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:38:54.953+0000 7f523e183ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:54.958 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:38:54.955+0000 7f523e183ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:54.960 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:38:54.956+0000 7f523e183ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:55.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-03-11T22:38:55.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:38:55.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:38:55.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:38:55.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:38:55.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:38:55.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:38:55.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:38:55.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:38:55.277 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:38:55.495 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:38:55.491+0000 7f523e183ec0 -1 Falling back to public interface 2020-03-11T22:38:55.622 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 45 up_thru 45 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/24171,v1:127.0.0.1:6819/24171] [v2:127.0.0.1:6820/24171,v1:127.0.0.1:6821/24171] exists,up 29aaf178-2506-493e-b1ca-ca8045da325e 2020-03-11T22:38:55.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:38:55.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:38:55.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:38:55.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:196: TEST_divergent: sleep 20 2020-03-11T22:38:56.367 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:38:56.363+0000 7f523e183ec0 -1 osd.1 46 log_to_monitors {default=true} 2020-03-11T22:39:15.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:198: TEST_divergent: echo 'allowing recovery' 2020-03-11T22:39:15.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:199: TEST_divergent: ceph pg dump pgs 2020-03-11T22:39:15.623 INFO:tasks.workunit.client.0.smithi145.stdout:allowing recovery 2020-03-11T22:39:15.954 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:39:15.954 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 101 0 0 0 0 48430 0 0 18 18 active+clean 2020-03-11T22:38:57.495725+0000 46'102 49:150 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:37:17.979060+0000 0'0 2020-03-11T22:37:17.979060+0000 0 2020-03-11T22:39:15.954 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:39:15.954 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:39:15.965 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:39:15.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2020-03-11T22:39:15.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.0 debug kick_recovery_wq 0 2020-03-11T22:39:16.086 INFO:tasks.workunit.client.0.smithi145.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:39:16.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2020-03-11T22:39:16.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.1 debug kick_recovery_wq 0 2020-03-11T22:39:16.214 INFO:tasks.workunit.client.0.smithi145.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:39:16.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2020-03-11T22:39:16.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.2 debug kick_recovery_wq 0 2020-03-11T22:39:16.342 INFO:tasks.workunit.client.0.smithi145.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:39:16.350 INFO:tasks.workunit.client.0.smithi145.stdout:reading divergent objects 2020-03-11T22:39:16.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:206: TEST_divergent: echo 'reading divergent objects' 2020-03-11T22:39:16.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:207: TEST_divergent: ceph pg dump pgs 2020-03-11T22:39:16.670 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:39:16.670 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 101 0 0 0 0 48430 0 0 18 18 active+clean 2020-03-11T22:38:57.495725+0000 46'102 49:150 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:37:17.979060+0000 0'0 2020-03-11T22:37:17.979060+0000 0 2020-03-11T22:39:16.670 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:39:16.670 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:39:16.672 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:39:16.680 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: expr 2 + 2 2020-03-11T22:39:16.681 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: seq 1 4 2020-03-11T22:39:16.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:39:16.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_1 td/divergent-priors/existing 2020-03-11T22:39:16.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:39:16.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_2 td/divergent-priors/existing 2020-03-11T22:39:16.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:39:16.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_3 td/divergent-priors/existing 2020-03-11T22:39:16.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:39:16.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_4 td/divergent-priors/existing 2020-03-11T22:39:16.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:212: TEST_divergent: rm -f td/divergent-priors/existing 2020-03-11T22:39:16.807 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: find td/divergent-priors -name '*osd*log' 2020-03-11T22:39:16.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:39:16.812 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:41.235+0000 7fcb8c96c700 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4522.0:1 2020-03-11T22:37:38.816083+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:39:16.812 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:41.235+0000 7fcb8c96c700 20 _merge_object_divergent_entries: keeping 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4522.0:1 2020-03-11T22:37:38.816083+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:39:16.812 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:41.235+0000 7fcb8c96c700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:39:16.812 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:41.235+0000 7fcb8c96c700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 29'100 first_divergent_update: 29'101 last_divergent_update: 29'101 2020-03-11T22:39:16.812 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:41.235+0000 7fcb8c96c700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2020-03-11T22:39:16.812 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:41.235+0000 7fcb8c96c700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:39:16.812 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:41.235+0000 7fcb8c96c700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 29'101 2020-03-11T22:39:16.813 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:41.235+0000 7fcb8c96c700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4522.0:1 2020-03-11T22:37:38.816083+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:39:16.813 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:38:41.235+0000 7fcb8c96c700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2020-03-11T22:39:16.813 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: find td/divergent-priors -name '*osd*log' 2020-03-11T22:39:16.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: grep -q '_merge_object_divergent_entries.*cannot roll back, removing and adding to missing' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:39:16.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:221: TEST_divergent: echo success 2020-03-11T22:39:16.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:223: TEST_divergent: delete_pool test 2020-03-11T22:39:16.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test 2020-03-11T22:39:16.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-03-11T22:39:16.817 INFO:tasks.workunit.client.0.smithi145.stdout:success 2020-03-11T22:39:17.330 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' does not exist 2020-03-11T22:39:17.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:224: TEST_divergent: kill_daemons td/divergent-priors 2020-03-11T22:39:17.338 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:39:17.338 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:39:17.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:39:17.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:39:17.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:39:17.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:39:17.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2020-03-11T22:39:17.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:39:17.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:39:17.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:39:17.446 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:39:17.447 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:39:17.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:39:17.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:39:17.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:39:17.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:39:17.450 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:39:17.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:39:17.451 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:39:17.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:39:17.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:39:17.452 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:39:17.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:39:17.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:39:17.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:39:17.455 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:39:17.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:39:17.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:39:17.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:39:17.467 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:39:17.467 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:17.467 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:39:17.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17402 2020-03-11T22:39:17.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:39:17.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:39:17.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2020-03-11T22:39:17.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2020-03-11T22:39:17.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2020-03-11T22:39:17.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2020-03-11T22:39:17.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:39:17.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:39:17.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:39:17.470 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:39:17.470 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:39:17.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:39:17.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:39:17.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:39:17.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:39:17.471 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:39:17.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:39:17.472 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:39:17.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:39:17.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:39:17.473 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:39:17.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:39:17.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:39:17.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:39:17.475 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:39:17.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:39:17.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:39:17.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:39:17.477 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:39:17.477 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:17.477 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:39:17.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17402 2020-03-11T22:39:17.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:39:17.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:39:17.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2020-03-11T22:39:17.479 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:39:17.479 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:17.479 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:39:17.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17402 2020-03-11T22:39:17.481 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:39:17.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-03-11T22:39:17.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:39:17.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2020-03-11T22:39:17.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_2 td/divergent-priors 2020-03-11T22:39:17.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:421: TEST_divergent_2: local dir=td/divergent-priors 2020-03-11T22:39:17.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:424: TEST_divergent_2: local dummyfile=/etc/fstab 2020-03-11T22:39:17.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:425: TEST_divergent_2: local dummyfile2=/etc/resolv.conf 2020-03-11T22:39:17.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:427: TEST_divergent_2: local num_osds=3 2020-03-11T22:39:17.484 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: expr 3 - 1 2020-03-11T22:39:17.484 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: seq 0 2 2020-03-11T22:39:17.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: local 'osds=0 2020-03-11T22:39:17.485 INFO:tasks.workunit.client.0.smithi145.stderr:1 2020-03-11T22:39:17.485 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2020-03-11T22:39:17.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:429: TEST_divergent_2: run_mon td/divergent-priors a 2020-03-11T22:39:17.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2020-03-11T22:39:17.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:39:17.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:39:17.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:39:17.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2020-03-11T22:39:17.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2020-03-11T22:39:17.678 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:39:17.678 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:39:17.678 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:39:17.679 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:39:17.679 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:17.679 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:39:17.679 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:39:17.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:39:17.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:39:17.861 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:39:17.861 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:39:17.861 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:39:17.861 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:39:17.861 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:39:17.862 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:39:17.862 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:39:17.862 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:39:17.862 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:39:17.862 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:17.862 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:39:17.862 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17402/ceph-mon.a.asok 2020-03-11T22:39:17.863 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:39:17.863 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17402/ceph-mon.a.asok config get fsid 2020-03-11T22:39:17.948 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:39:17.948 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:39:17.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:39:17.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:39:17.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:39:17.949 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:39:17.949 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:39:17.949 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:39:17.949 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:39:17.949 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:17.950 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:39:17.950 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17402/ceph-mon.a.asok 2020-03-11T22:39:17.950 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:39:17.950 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17402/ceph-mon.a.asok config get mon_host 2020-03-11T22:39:18.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:430: TEST_divergent_2: run_mgr td/divergent-priors x 2020-03-11T22:39:18.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/divergent-priors 2020-03-11T22:39:18.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:39:18.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:39:18.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:39:18.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/divergent-priors/x 2020-03-11T22:39:18.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:39:18.532 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:39:18.533 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:39:18.533 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:39:18.533 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:39:18.533 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:18.533 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:39:18.533 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:39:18.534 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:39:18.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:39:18.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2020-03-11T22:39:18.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 0 2020-03-11T22:39:18.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:39:18.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:39:18.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:39:18.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:39:18.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:39:18.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:39:18.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:39:18.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:39:18.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:39:18.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:39:18.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:39:18.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:39:18.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:39:18.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:39:18.566 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:39:18.566 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:39:18.566 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:39:18.566 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:39:18.566 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:18.566 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:39:18.566 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:39:18.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:39:18.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:39:18.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:39:18.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:39:18.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:39:18.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:39:18.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:39:18.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:39:18.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:39:18.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:39:18.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:39:18.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:39:18.568 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:39:18.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=6548d5c2-3fe9-45a6-8404-614e80715021 2020-03-11T22:39:18.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 6548d5c2-3fe9-45a6-8404-614e80715021' 2020-03-11T22:39:18.568 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:39:18.569 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 6548d5c2-3fe9-45a6-8404-614e80715021 2020-03-11T22:39:18.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCWaGleAbVzIhAAM/0fe5jT0rdmE94ctJqXXQ== 2020-03-11T22:39:18.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCWaGleAbVzIhAAM/0fe5jT0rdmE94ctJqXXQ=="}' 2020-03-11T22:39:18.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 6548d5c2-3fe9-45a6-8404-614e80715021 -i td/divergent-priors/0/new.json 2020-03-11T22:39:18.906 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:39:18.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/0/new.json 2020-03-11T22:39:18.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCWaGleAbVzIhAAM/0fe5jT0rdmE94ctJqXXQ== --osd-uuid 6548d5c2-3fe9-45a6-8404-614e80715021 2020-03-11T22:39:18.945 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:39:18.942+0000 7fc07c69bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:18.947 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:39:18.944+0000 7fc07c69bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:18.947 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:39:18.945+0000 7fc07c69bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:18.948 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:39:18.945+0000 7fc07c69bec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:39:18.948 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:39:18.945+0000 7fc07c69bec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:39:18.948 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:39:18.945+0000 7fc07c69bec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:39:18.991 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:39:18.989+0000 7fc07c69bec0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2020-03-11T22:39:21.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/0/keyring 2020-03-11T22:39:21.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:39:21.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:39:21.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:39:21.938 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2020-03-11T22:39:22.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:39:22.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:39:22.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:39:22.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:39:22.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:39:22.399 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2020-03-11T22:39:22.425 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:39:22.423+0000 7f685be8dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:22.427 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:39:22.425+0000 7f685be8dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:22.429 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:39:22.427+0000 7f685be8dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:22.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:39:22.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:39:22.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:39:22.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:39:22.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:39:22.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:39:22.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:39:22.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:39:22.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:39:22.747 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:39:22.945 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:39:22.943+0000 7f685be8dec0 -1 Falling back to public interface 2020-03-11T22:39:23.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:39:23.675 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:39:23.673+0000 7f685be8dec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:39:24.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:39:24.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:39:24.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:39:24.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:39:24.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:39:24.097 INFO:tasks.workunit.client.0.smithi145.stdout:1 2020-03-11T22:39:24.506 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/25473,v1:127.0.0.1:6803/25473] [v2:127.0.0.1:6804/25473,v1:127.0.0.1:6805/25473] exists,up 6548d5c2-3fe9-45a6-8404-614e80715021 2020-03-11T22:39:24.506 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 fc159c74-e2d8-44ac-b32a-c54afc816878 2020-03-11T22:39:24.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:39:24.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:39:24.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:39:24.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2020-03-11T22:39:24.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 1 2020-03-11T22:39:24.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:39:24.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:39:24.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:39:24.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:39:24.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:39:24.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:39:24.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:39:24.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:39:24.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:39:24.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:39:24.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:39:24.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:39:24.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:39:24.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:39:24.510 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:39:24.510 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:39:24.510 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:39:24.510 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:39:24.510 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:24.510 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:39:24.510 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:39:24.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:39:24.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:39:24.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:39:24.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:39:24.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:39:24.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:39:24.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:39:24.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:39:24.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:39:24.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:39:24.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:39:24.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:39:24.512 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:39:24.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=fc159c74-e2d8-44ac-b32a-c54afc816878 2020-03-11T22:39:24.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 fc159c74-e2d8-44ac-b32a-c54afc816878' 2020-03-11T22:39:24.513 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:39:24.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCcaGler8wSHxAAz83znuj1cgBiUhQWJfraAQ== 2020-03-11T22:39:24.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCcaGler8wSHxAAz83znuj1cgBiUhQWJfraAQ=="}' 2020-03-11T22:39:24.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new fc159c74-e2d8-44ac-b32a-c54afc816878 -i td/divergent-priors/1/new.json 2020-03-11T22:39:24.970 INFO:tasks.workunit.client.0.smithi145.stdout:1 2020-03-11T22:39:24.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/1/new.json 2020-03-11T22:39:24.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCcaGler8wSHxAAz83znuj1cgBiUhQWJfraAQ== --osd-uuid fc159c74-e2d8-44ac-b32a-c54afc816878 2020-03-11T22:39:25.008 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:39:25.006+0000 7f5d0198dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:25.032 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:39:25.029+0000 7f5d0198dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:25.033 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:39:25.030+0000 7f5d0198dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:25.033 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:39:25.031+0000 7f5d0198dec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:39:25.034 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:39:25.031+0000 7f5d0198dec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:39:25.034 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:39:25.031+0000 7f5d0198dec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:39:25.107 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:39:25.104+0000 7f5d0198dec0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2020-03-11T22:39:28.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/1/keyring 2020-03-11T22:39:28.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:39:28.139 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2020-03-11T22:39:28.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:39:28.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:39:28.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:39:28.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:39:28.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:39:28.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:39:28.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:39:28.582 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2020-03-11T22:39:28.608 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:39:28.605+0000 7f5f34958ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:28.609 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:39:28.607+0000 7f5f34958ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:28.623 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:39:28.608+0000 7f5f34958ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:28.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:39:28.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:39:28.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:39:28.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:39:28.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:39:28.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:39:28.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:39:28.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:39:28.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:39:28.926 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:39:29.126 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:39:29.124+0000 7f5f34958ec0 -1 Falling back to public interface 2020-03-11T22:39:29.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:39:29.867 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:39:29.864+0000 7f5f34958ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:39:30.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:39:30.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:39:30.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:39:30.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:39:30.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:39:30.278 INFO:tasks.workunit.client.0.smithi145.stdout:1 2020-03-11T22:39:30.641 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/25828,v1:127.0.0.1:6811/25828] [v2:127.0.0.1:6812/25828,v1:127.0.0.1:6813/25828] exists,up fc159c74-e2d8-44ac-b32a-c54afc816878 2020-03-11T22:39:30.641 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 cc077be5-2bbd-488f-9867-ebfa384fe3f6 2020-03-11T22:39:30.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:39:30.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:39:30.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:39:30.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2020-03-11T22:39:30.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 2 2020-03-11T22:39:30.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:39:30.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:39:30.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:39:30.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:39:30.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:39:30.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:39:30.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:39:30.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:39:30.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:39:30.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:39:30.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:39:30.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:39:30.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:39:30.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:39:30.645 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:39:30.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:39:30.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:39:30.646 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:39:30.646 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:30.646 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:39:30.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:39:30.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:39:30.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:39:30.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:39:30.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:39:30.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:39:30.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:39:30.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:39:30.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:39:30.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:39:30.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:39:30.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:39:30.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:39:30.648 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:39:30.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=cc077be5-2bbd-488f-9867-ebfa384fe3f6 2020-03-11T22:39:30.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 cc077be5-2bbd-488f-9867-ebfa384fe3f6' 2020-03-11T22:39:30.648 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:39:30.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCiaGleaPNDJhAA75BDZNjkXCL9aCq0Hfi94A== 2020-03-11T22:39:30.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCiaGleaPNDJhAA75BDZNjkXCL9aCq0Hfi94A=="}' 2020-03-11T22:39:30.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new cc077be5-2bbd-488f-9867-ebfa384fe3f6 -i td/divergent-priors/2/new.json 2020-03-11T22:39:31.152 INFO:tasks.workunit.client.0.smithi145.stdout:2 2020-03-11T22:39:31.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/2/new.json 2020-03-11T22:39:31.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCiaGleaPNDJhAA75BDZNjkXCL9aCq0Hfi94A== --osd-uuid cc077be5-2bbd-488f-9867-ebfa384fe3f6 2020-03-11T22:39:31.192 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:39:31.189+0000 7fe91b5bdec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:31.212 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:39:31.209+0000 7fe91b5bdec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:31.214 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:39:31.211+0000 7fe91b5bdec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:31.215 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:39:31.212+0000 7fe91b5bdec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:39:31.215 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:39:31.212+0000 7fe91b5bdec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:39:31.215 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:39:31.212+0000 7fe91b5bdec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:39:31.278 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:39:31.276+0000 7fe91b5bdec0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2020-03-11T22:39:34.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/2/keyring 2020-03-11T22:39:34.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:39:34.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:39:34.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:39:34.355 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2020-03-11T22:39:34.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:39:34.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:39:34.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:39:34.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:39:34.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:39:34.811 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2020-03-11T22:39:34.837 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:39:34.833+0000 7f57e3ed5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:34.839 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:39:34.835+0000 7f57e3ed5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:34.840 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:39:34.836+0000 7f57e3ed5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:35.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:39:35.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:39:35.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:39:35.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:39:35.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:39:35.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:39:35.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:39:35.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:39:35.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:39:35.163 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:39:35.355 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:39:35.352+0000 7f57e3ed5ec0 -1 Falling back to public interface 2020-03-11T22:39:35.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:39:36.081 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:39:36.077+0000 7f57e3ed5ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:39:36.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:39:36.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:39:36.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:39:36.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:39:36.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:39:36.513 INFO:tasks.workunit.client.0.smithi145.stdout:1 2020-03-11T22:39:36.862 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/26183,v1:127.0.0.1:6819/26183] [v2:127.0.0.1:6820/26183,v1:127.0.0.1:6821/26183] exists,up cc077be5-2bbd-488f-9867-ebfa384fe3f6 2020-03-11T22:39:36.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:39:36.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:39:36.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:39:36.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:436: TEST_divergent_2: ceph osd set noout 2020-03-11T22:39:37.476 INFO:tasks.workunit.client.0.smithi145.stderr:noout is set 2020-03-11T22:39:37.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:437: TEST_divergent_2: ceph osd set noin 2020-03-11T22:39:38.224 INFO:tasks.workunit.client.0.smithi145.stderr:noin is set 2020-03-11T22:39:38.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:438: TEST_divergent_2: ceph osd set nodown 2020-03-11T22:39:38.879 INFO:tasks.workunit.client.0.smithi145.stderr:nodown is set 2020-03-11T22:39:38.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:439: TEST_divergent_2: create_pool test 1 1 2020-03-11T22:39:38.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-03-11T22:39:39.401 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' already exists 2020-03-11T22:39:39.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:39:40.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:440: TEST_divergent_2: ceph osd pool set test size 3 2020-03-11T22:39:41.107 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 3 2020-03-11T22:39:41.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:441: TEST_divergent_2: ceph osd pool set test min_size 2 2020-03-11T22:39:41.683 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 min_size to 2 2020-03-11T22:39:41.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:443: TEST_divergent_2: flush_pg_stats 2020-03-11T22:39:41.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:39:41.692 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:39:42.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:39:42.056 INFO:tasks.workunit.client.0.smithi145.stderr:1 2020-03-11T22:39:42.056 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2020-03-11T22:39:42.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:39:42.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:42.057 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:39:42.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836486 2020-03-11T22:39:42.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486' 2020-03-11T22:39:42.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:42.185 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:39:42.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672965 2020-03-11T22:39:42.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672965' 2020-03-11T22:39:42.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:42.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:39:42.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509444 2020-03-11T22:39:42.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672965 2-64424509444' 2020-03-11T22:39:42.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:42.444 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836486 2020-03-11T22:39:42.444 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:42.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:39:42.445 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836486 2020-03-11T22:39:42.445 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:42.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836486 2020-03-11T22:39:42.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2020-03-11T22:39:42.446 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:39:42.446 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836486 2020-03-11T22:39:42.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836484 -lt 21474836486 2020-03-11T22:39:42.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:39:43.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:39:43.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:39:44.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836486 -lt 21474836486 2020-03-11T22:39:44.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:44.141 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672965 2020-03-11T22:39:44.141 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:44.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:39:44.142 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672965 2020-03-11T22:39:44.142 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:44.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672965 2020-03-11T22:39:44.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2020-03-11T22:39:44.143 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:39:44.143 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672965 2020-03-11T22:39:44.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672965 -lt 42949672965 2020-03-11T22:39:44.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:44.515 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509444 2020-03-11T22:39:44.515 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:44.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:39:44.516 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509444 2020-03-11T22:39:44.516 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:44.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509444 2020-03-11T22:39:44.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2020-03-11T22:39:44.517 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:39:44.518 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509444 2020-03-11T22:39:44.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509444 -lt 64424509444 2020-03-11T22:39:44.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:444: TEST_divergent_2: wait_for_clean 2020-03-11T22:39:44.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:39:44.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:39:44.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:39:44.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:39:44.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:39:44.859 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:39:44.859 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:39:44.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:39:44.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:39:44.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:39:44.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:39:44.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:39:44.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:39:44.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:39:44.950 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:39:45.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:39:45.294 INFO:tasks.workunit.client.0.smithi145.stderr:1 2020-03-11T22:39:45.294 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2020-03-11T22:39:45.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:39:45.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:45.294 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:39:45.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836489 2020-03-11T22:39:45.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489' 2020-03-11T22:39:45.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:45.423 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:39:45.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672967 2020-03-11T22:39:45.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-42949672967' 2020-03-11T22:39:45.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:45.552 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:39:45.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509446 2020-03-11T22:39:45.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-42949672967 2-64424509446' 2020-03-11T22:39:45.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:45.682 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836489 2020-03-11T22:39:45.682 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:45.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:39:45.683 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836489 2020-03-11T22:39:45.683 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:45.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836489 2020-03-11T22:39:45.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2020-03-11T22:39:45.684 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:39:45.685 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836489 2020-03-11T22:39:46.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836489 -lt 21474836489 2020-03-11T22:39:46.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:46.028 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672967 2020-03-11T22:39:46.028 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:46.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:39:46.029 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672967 2020-03-11T22:39:46.029 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:46.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672967 2020-03-11T22:39:46.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2020-03-11T22:39:46.030 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:39:46.031 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672967 2020-03-11T22:39:46.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672967 -lt 42949672967 2020-03-11T22:39:46.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:46.377 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509446 2020-03-11T22:39:46.377 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:46.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:39:46.378 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509446 2020-03-11T22:39:46.378 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:46.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509446 2020-03-11T22:39:46.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2020-03-11T22:39:46.380 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:39:46.380 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509446 2020-03-11T22:39:46.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509446 -lt 64424509446 2020-03-11T22:39:46.733 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:39:46.733 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:46.734 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:47.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:39:47.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:47.176 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:47.176 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:47.176 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:47.176 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:47.176 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:47.177 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:47.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:39:47.506 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:47.507 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:47.507 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:47.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:39:47.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:39:47.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:39:47.957 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: ceph pg dump pgs --format=json 2020-03-11T22:39:47.958 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: jq '.pg_stats[0].up_primary' 2020-03-11T22:39:48.282 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:39:48.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: local divergent=1 2020-03-11T22:39:48.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:448: TEST_divergent_2: echo 'primary and soon to be divergent is 1' 2020-03-11T22:39:48.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:449: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:39:48.291 INFO:tasks.workunit.client.0.smithi145.stdout:primary and soon to be divergent is 1 2020-03-11T22:39:48.613 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:39:48.613 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:39:40.599360+0000 0'0 26:12 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:39.216542+0000 0'0 2020-03-11T22:39:39.216542+0000 0 2020-03-11T22:39:48.614 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:39:48.614 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:39:48.615 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:39:48.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:450: TEST_divergent_2: local non_divergent= 2020-03-11T22:39:48.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2020-03-11T22:39:48.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 0 = 1 ']' 2020-03-11T22:39:48.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0' 2020-03-11T22:39:48.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2020-03-11T22:39:48.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 1 = 1 ']' 2020-03-11T22:39:48.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:454: TEST_divergent_2: continue 2020-03-11T22:39:48.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2020-03-11T22:39:48.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 2 = 1 ']' 2020-03-11T22:39:48.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0 2' 2020-03-11T22:39:48.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:459: TEST_divergent_2: echo 'writing initial objects' 2020-03-11T22:39:48.626 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: seq 1 100 2020-03-11T22:39:48.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:48.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_1 /etc/fstab 2020-03-11T22:39:48.626 INFO:tasks.workunit.client.0.smithi145.stdout:writing initial objects 2020-03-11T22:39:48.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:48.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_2 /etc/fstab 2020-03-11T22:39:48.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:48.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_3 /etc/fstab 2020-03-11T22:39:48.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:48.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_4 /etc/fstab 2020-03-11T22:39:48.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:48.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_5 /etc/fstab 2020-03-11T22:39:49.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_6 /etc/fstab 2020-03-11T22:39:49.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_7 /etc/fstab 2020-03-11T22:39:49.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_8 /etc/fstab 2020-03-11T22:39:49.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_9 /etc/fstab 2020-03-11T22:39:49.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_10 /etc/fstab 2020-03-11T22:39:49.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_11 /etc/fstab 2020-03-11T22:39:49.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_12 /etc/fstab 2020-03-11T22:39:49.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_13 /etc/fstab 2020-03-11T22:39:49.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_14 /etc/fstab 2020-03-11T22:39:49.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_15 /etc/fstab 2020-03-11T22:39:49.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_16 /etc/fstab 2020-03-11T22:39:49.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_17 /etc/fstab 2020-03-11T22:39:49.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_18 /etc/fstab 2020-03-11T22:39:49.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_19 /etc/fstab 2020-03-11T22:39:49.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:49.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_20 /etc/fstab 2020-03-11T22:39:50.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:50.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_21 /etc/fstab 2020-03-11T22:39:50.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:50.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_22 /etc/fstab 2020-03-11T22:39:50.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:50.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_23 /etc/fstab 2020-03-11T22:39:50.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:50.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_24 /etc/fstab 2020-03-11T22:39:50.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:50.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_25 /etc/fstab 2020-03-11T22:39:50.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:50.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_26 /etc/fstab 2020-03-11T22:39:50.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:50.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_27 /etc/fstab 2020-03-11T22:39:50.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:50.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_28 /etc/fstab 2020-03-11T22:39:50.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:50.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_29 /etc/fstab 2020-03-11T22:39:50.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:50.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_30 /etc/fstab 2020-03-11T22:39:50.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:50.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_31 /etc/fstab 2020-03-11T22:39:50.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:50.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_32 /etc/fstab 2020-03-11T22:39:50.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:50.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_33 /etc/fstab 2020-03-11T22:39:50.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:50.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_34 /etc/fstab 2020-03-11T22:39:50.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:50.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_35 /etc/fstab 2020-03-11T22:39:51.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:51.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_36 /etc/fstab 2020-03-11T22:39:51.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:51.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_37 /etc/fstab 2020-03-11T22:39:51.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:51.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_38 /etc/fstab 2020-03-11T22:39:51.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:51.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_39 /etc/fstab 2020-03-11T22:39:51.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:51.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_40 /etc/fstab 2020-03-11T22:39:51.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:51.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_41 /etc/fstab 2020-03-11T22:39:51.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:51.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_42 /etc/fstab 2020-03-11T22:39:51.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:51.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_43 /etc/fstab 2020-03-11T22:39:51.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:51.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_44 /etc/fstab 2020-03-11T22:39:51.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:51.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_45 /etc/fstab 2020-03-11T22:39:51.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:51.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_46 /etc/fstab 2020-03-11T22:39:51.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:51.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_47 /etc/fstab 2020-03-11T22:39:51.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:51.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_48 /etc/fstab 2020-03-11T22:39:51.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:51.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_49 /etc/fstab 2020-03-11T22:39:51.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:51.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_50 /etc/fstab 2020-03-11T22:39:52.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:52.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_51 /etc/fstab 2020-03-11T22:39:52.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:52.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_52 /etc/fstab 2020-03-11T22:39:52.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:52.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_53 /etc/fstab 2020-03-11T22:39:52.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:52.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_54 /etc/fstab 2020-03-11T22:39:52.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:52.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_55 /etc/fstab 2020-03-11T22:39:52.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:52.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_56 /etc/fstab 2020-03-11T22:39:52.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:52.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_57 /etc/fstab 2020-03-11T22:39:52.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:52.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_58 /etc/fstab 2020-03-11T22:39:52.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:52.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_59 /etc/fstab 2020-03-11T22:39:52.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:52.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_60 /etc/fstab 2020-03-11T22:39:52.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:52.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_61 /etc/fstab 2020-03-11T22:39:52.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:52.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_62 /etc/fstab 2020-03-11T22:39:52.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:52.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_63 /etc/fstab 2020-03-11T22:39:52.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:52.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_64 /etc/fstab 2020-03-11T22:39:52.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:52.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_65 /etc/fstab 2020-03-11T22:39:52.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:52.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_66 /etc/fstab 2020-03-11T22:39:53.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:53.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_67 /etc/fstab 2020-03-11T22:39:53.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:53.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_68 /etc/fstab 2020-03-11T22:39:53.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:53.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_69 /etc/fstab 2020-03-11T22:39:53.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:53.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_70 /etc/fstab 2020-03-11T22:39:53.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:53.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_71 /etc/fstab 2020-03-11T22:39:53.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:53.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_72 /etc/fstab 2020-03-11T22:39:53.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:53.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_73 /etc/fstab 2020-03-11T22:39:53.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:53.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_74 /etc/fstab 2020-03-11T22:39:53.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:53.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_75 /etc/fstab 2020-03-11T22:39:53.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:53.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_76 /etc/fstab 2020-03-11T22:39:53.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:53.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_77 /etc/fstab 2020-03-11T22:39:53.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:53.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_78 /etc/fstab 2020-03-11T22:39:53.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:53.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_79 /etc/fstab 2020-03-11T22:39:54.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:54.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_80 /etc/fstab 2020-03-11T22:39:54.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:54.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_81 /etc/fstab 2020-03-11T22:39:54.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:54.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_82 /etc/fstab 2020-03-11T22:39:54.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:54.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_83 /etc/fstab 2020-03-11T22:39:54.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:54.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_84 /etc/fstab 2020-03-11T22:39:54.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:54.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_85 /etc/fstab 2020-03-11T22:39:54.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:54.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_86 /etc/fstab 2020-03-11T22:39:54.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:54.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_87 /etc/fstab 2020-03-11T22:39:54.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:54.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_88 /etc/fstab 2020-03-11T22:39:54.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:54.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_89 /etc/fstab 2020-03-11T22:39:54.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:54.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_90 /etc/fstab 2020-03-11T22:39:54.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:54.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_91 /etc/fstab 2020-03-11T22:39:54.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:54.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_92 /etc/fstab 2020-03-11T22:39:54.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:54.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_93 /etc/fstab 2020-03-11T22:39:54.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:54.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_94 /etc/fstab 2020-03-11T22:39:55.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:55.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_95 /etc/fstab 2020-03-11T22:39:55.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:55.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_96 /etc/fstab 2020-03-11T22:39:55.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:55.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_97 /etc/fstab 2020-03-11T22:39:55.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:55.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_98 /etc/fstab 2020-03-11T22:39:55.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:55.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_99 /etc/fstab 2020-03-11T22:39:55.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:39:55.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_100 /etc/fstab 2020-03-11T22:39:55.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:39:55.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: wait_for_clean 2020-03-11T22:39:55.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:39:55.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:39:55.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:39:55.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:39:55.439 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:39:55.439 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:39:55.439 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:39:55.439 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:39:55.439 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:39:55.439 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:39:55.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:39:55.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:39:55.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:39:55.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:39:55.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:39:55.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:39:55.835 INFO:tasks.workunit.client.0.smithi145.stderr:1 2020-03-11T22:39:55.835 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2020-03-11T22:39:55.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:39:55.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:55.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:39:55.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836493 2020-03-11T22:39:55.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493' 2020-03-11T22:39:55.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:55.964 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:39:56.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672971 2020-03-11T22:39:56.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672971' 2020-03-11T22:39:56.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:56.095 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:39:56.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509450 2020-03-11T22:39:56.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672971 2-64424509450' 2020-03-11T22:39:56.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:56.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836493 2020-03-11T22:39:56.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:56.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:39:56.225 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836493 2020-03-11T22:39:56.225 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:56.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836493 2020-03-11T22:39:56.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2020-03-11T22:39:56.226 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:39:56.226 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836493 2020-03-11T22:39:56.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836491 -lt 21474836493 2020-03-11T22:39:56.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:39:57.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:39:57.567 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:39:57.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836493 -lt 21474836493 2020-03-11T22:39:57.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:57.934 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672971 2020-03-11T22:39:57.934 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:57.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:39:57.935 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672971 2020-03-11T22:39:57.936 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:57.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672971 2020-03-11T22:39:57.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2020-03-11T22:39:57.937 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:39:57.937 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672971 2020-03-11T22:39:58.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672972 -lt 42949672971 2020-03-11T22:39:58.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:58.301 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509450 2020-03-11T22:39:58.301 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:58.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:39:58.302 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509450 2020-03-11T22:39:58.303 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:58.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509450 2020-03-11T22:39:58.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2020-03-11T22:39:58.304 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:39:58.304 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509450 2020-03-11T22:39:58.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509451 -lt 64424509450 2020-03-11T22:39:58.648 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:39:58.649 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:58.649 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:59.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:39:59.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:59.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:59.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:59.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:59.132 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:59.132 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:59.132 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:59.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:39:59.474 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:59.474 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:59.474 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:59.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:39:59.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:39:59.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:39:59.912 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: get_pg test existing_1 2020-03-11T22:39:59.912 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local poolname=test 2020-03-11T22:39:59.913 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local objectname=existing_1 2020-03-11T22:39:59.913 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: ceph --format json osd map test existing_1 2020-03-11T22:39:59.913 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: jq -r .pgid 2020-03-11T22:40:00.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: local pgid=1.0 2020-03-11T22:40:00.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:471: TEST_divergent_2: echo 'blackholing osds 0 2' 2020-03-11T22:40:00.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:472: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:40:00.291 INFO:tasks.workunit.client.0.smithi145.stdout:blackholing osds 0 2 2020-03-11T22:40:00.611 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:40:00.611 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2020-03-11T22:39:40.599360+0000 28'100 28:112 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:39.216542+0000 0'0 2020-03-11T22:39:39.216542+0000 0 2020-03-11T22:40:00.612 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:40:00.612 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:40:00.613 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:40:00.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:40:00.622 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.0 2020-03-11T22:40:00.622 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:40:00.622 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:40:00.622 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:40:00.623 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:00.623 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:40:00.623 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17402/ceph-osd.0.asok 2020-03-11T22:40:00.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:40:00.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17402/ceph-osd.0.asok config set objectstore_blackhole 1 2020-03-11T22:40:00.708 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:40:00.708 INFO:tasks.workunit.client.0.smithi145.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:40:00.709 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:40:00.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:40:00.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.2 2020-03-11T22:40:00.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:40:00.717 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:40:00.717 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:40:00.717 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:00.717 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:40:00.717 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17402/ceph-osd.2.asok 2020-03-11T22:40:00.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:40:00.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17402/ceph-osd.2.asok config set objectstore_blackhole 1 2020-03-11T22:40:00.804 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:40:00.804 INFO:tasks.workunit.client.0.smithi145.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:40:00.804 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:40:00.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:479: TEST_divergent_2: echo 'create new divergent objects' 2020-03-11T22:40:00.811 INFO:tasks.workunit.client.0.smithi145.stdout:create new divergent objects 2020-03-11T22:40:00.811 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: seq 1 2 2020-03-11T22:40:00.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:40:00.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:40:00.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_1 2020-03-11T22:40:00.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:485: TEST_divergent_2: echo 'writing divergent objects' 2020-03-11T22:40:00.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_2 2020-03-11T22:40:00.813 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: seq 1 2 2020-03-11T22:40:00.813 INFO:tasks.workunit.client.0.smithi145.stdout:writing divergent objects 2020-03-11T22:40:00.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2020-03-11T22:40:00.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2020-03-11T22:40:00.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_1 /etc/resolv.conf 2020-03-11T22:40:00.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:491: TEST_divergent_2: echo 'remove divergent objects' 2020-03-11T22:40:00.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_2 /etc/resolv.conf 2020-03-11T22:40:00.815 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: seq 1 2 2020-03-11T22:40:00.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-03-11T22:40:00.815 INFO:tasks.workunit.client.0.smithi145.stdout:remove divergent objects 2020-03-11T22:40:00.816 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 1 + 2 2020-03-11T22:40:00.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=3 2020-03-11T22:40:00.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-03-11T22:40:00.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_3 2020-03-11T22:40:00.816 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 2 + 2 2020-03-11T22:40:00.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=4 2020-03-11T22:40:00.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:497: TEST_divergent_2: sleep 10 2020-03-11T22:40:00.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_4 2020-03-11T22:40:10.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:498: TEST_divergent_2: killall -9 rados 2020-03-11T22:40:10.822 INFO:tasks.workunit.client.0.smithi145.stderr:rados: no process found 2020-03-11T22:40:10.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:501: TEST_divergent_2: echo 'killing all the osds' 2020-03-11T22:40:10.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:502: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:40:10.822 INFO:tasks.workunit.client.0.smithi145.stdout:killing all the osds 2020-03-11T22:40:11.150 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:40:11.150 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 active+clean 2020-03-11T22:39:40.599360+0000 28'106 28:118 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:39.216542+0000 0'0 2020-03-11T22:39:39.216542+0000 0 2020-03-11T22:40:11.150 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:40:11.151 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:40:11.152 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:40:11.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:503: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd 2020-03-11T22:40:11.160 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:40:11.161 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:40:11.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:40:11.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:40:11.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:40:11.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:40:11.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2020-03-11T22:40:11.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.0 2020-03-11T22:40:11.738 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already down. 2020-03-11T22:40:11.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2020-03-11T22:40:11.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.1 2020-03-11T22:40:12.204 INFO:tasks.workunit.client.0.smithi145.stderr:osd.1 is already down. 2020-03-11T22:40:12.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2020-03-11T22:40:12.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.2 2020-03-11T22:40:12.672 INFO:tasks.workunit.client.0.smithi145.stderr:osd.2 is already down. 2020-03-11T22:40:12.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:40:12.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.0 2020-03-11T22:40:13.141 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already out. 2020-03-11T22:40:13.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:40:13.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.2 2020-03-11T22:40:13.665 INFO:tasks.workunit.client.0.smithi145.stderr:osd.2 is already out. 2020-03-11T22:40:13.673 INFO:tasks.workunit.client.0.smithi145.stdout:bringing up non_divergent 0 2 2020-03-11T22:40:13.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:514: TEST_divergent_2: echo 'bringing up non_divergent 0 2' 2020-03-11T22:40:13.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:515: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:40:13.992 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:40:13.992 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 stale+active+clean 2020-03-11T22:39:40.599360+0000 28'106 28:118 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:39.216542+0000 0'0 2020-03-11T22:39:39.216542+0000 0 2020-03-11T22:40:13.992 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:40:13.993 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:40:13.994 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:40:14.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:40:14.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 0 2020-03-11T22:40:14.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:40:14.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:40:14.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:40:14.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:40:14.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:40:14.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:40:14.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:40:14.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:40:14.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:40:14.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:40:14.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:40:14.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:40:14.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:40:14.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:40:14.013 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:40:14.013 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:40:14.014 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:40:14.014 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:40:14.014 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:14.014 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:40:14.014 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:40:14.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:40:14.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:40:14.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:40:14.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:40:14.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:40:14.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:40:14.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:40:14.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:40:14.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:40:14.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:40:14.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:40:14.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:40:14.018 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-03-11T22:40:14.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:40:14.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:40:14.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:40:14.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:40:14.019 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2020-03-11T22:40:14.031 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:40:14.028+0000 7fa474207ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:14.033 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:40:14.030+0000 7fa474207ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:14.034 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:40:14.031+0000 7fa474207ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:14.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:40:14.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:40:14.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:40:14.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:40:14.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:40:14.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:14.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:40:14.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:14.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:40:14.359 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:40:14.548 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:40:14.545+0000 7fa474207ec0 -1 Falling back to public interface 2020-03-11T22:40:14.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:40:15.415 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:40:15.412+0000 7fa474207ec0 -1 osd.0 28 log_to_monitors {default=true} 2020-03-11T22:40:15.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:40:15.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:15.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:40:15.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:15.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:40:15.714 INFO:tasks.workunit.client.0.smithi145.stdout:1 2020-03-11T22:40:16.081 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up out weight 0 up_from 34 up_thru 0 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/30195,v1:127.0.0.1:6803/30195] [v2:127.0.0.1:6804/30195,v1:127.0.0.1:6805/30195] exists,up 6548d5c2-3fe9-45a6-8404-614e80715021 2020-03-11T22:40:16.082 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2020-03-11T22:40:16.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:40:16.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:40:16.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:40:16.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:40:16.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 2 2020-03-11T22:40:16.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:40:16.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:40:16.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:40:16.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:40:16.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:40:16.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:40:16.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:40:16.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:40:16.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:40:16.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:40:16.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:40:16.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:40:16.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:40:16.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:40:16.084 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:40:16.085 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:40:16.085 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:40:16.085 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:40:16.085 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:16.085 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:40:16.085 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:40:16.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:40:16.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:40:16.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:40:16.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:40:16.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:40:16.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:40:16.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:40:16.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:40:16.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:40:16.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:40:16.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:40:16.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:40:16.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-03-11T22:40:16.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:40:16.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:40:16.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:40:16.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:40:16.107 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:40:16.105+0000 7f032f57dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:16.109 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:40:16.106+0000 7f032f57dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:16.110 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:40:16.108+0000 7f032f57dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:16.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:40:16.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:40:16.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:40:16.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:40:16.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:40:16.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:16.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:40:16.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:16.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:40:16.448 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:40:16.626 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:40:16.624+0000 7f032f57dec0 -1 Falling back to public interface 2020-03-11T22:40:16.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:40:17.479 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:40:17.476+0000 7f032f57dec0 -1 osd.2 28 log_to_monitors {default=true} 2020-03-11T22:40:17.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:40:17.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:17.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:40:17.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:17.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:40:17.796 INFO:tasks.workunit.client.0.smithi145.stdout:1 2020-03-11T22:40:18.144 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up out weight 0 up_from 35 up_thru 0 down_at 31 last_clean_interval [15,28) [v2:127.0.0.1:6810/30389,v1:127.0.0.1:6811/30389] [v2:127.0.0.1:6812/30389,v1:127.0.0.1:6813/30389] exists,up cc077be5-2bbd-488f-9867-ebfa384fe3f6 2020-03-11T22:40:18.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:40:18.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:40:18.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:40:18.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:40:18.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.0 2020-03-11T22:40:18.713 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already in. 2020-03-11T22:40:18.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:40:18.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.2 2020-03-11T22:40:19.207 INFO:tasks.workunit.client.0.smithi145.stderr:osd.2 is already in. 2020-03-11T22:40:19.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:40:19.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: wait_for_clean 2020-03-11T22:40:19.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:40:19.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:40:19.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:40:19.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:40:19.217 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:40:19.217 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:40:19.218 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:40:19.218 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:40:19.218 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:40:19.218 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:40:19.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:40:19.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:40:19.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:40:19.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:40:19.274 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:40:19.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:40:19.691 INFO:tasks.workunit.client.0.smithi145.stderr:1 2020-03-11T22:40:19.691 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2020-03-11T22:40:19.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:40:19.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:40:19.691 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:40:19.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=146028888067 2020-03-11T22:40:19.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888067' 2020-03-11T22:40:19.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:40:19.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:40:19.911 INFO:tasks.workunit.client.0.smithi145.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:40:19.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:40:19.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888067 1-' 2020-03-11T22:40:19.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:40:19.919 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:40:20.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=150323855363 2020-03-11T22:40:20.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888067 1- 2-150323855363' 2020-03-11T22:40:20.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:40:20.049 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-146028888067 2020-03-11T22:40:20.049 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:40:20.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:40:20.050 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-146028888067 2020-03-11T22:40:20.051 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:40:20.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=146028888067 2020-03-11T22:40:20.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 146028888067' 2020-03-11T22:40:20.051 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:40:20.051 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 146028888067 2020-03-11T22:40:20.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 146028888067 2020-03-11T22:40:20.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:40:21.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:40:21.395 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:40:21.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 146028888067 2020-03-11T22:40:21.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:40:22.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 299 -eq 0 ']' 2020-03-11T22:40:22.748 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:40:23.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 146028888068 -lt 146028888067 2020-03-11T22:40:23.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:40:23.095 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:40:23.095 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:40:23.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:40:23.096 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:40:23.097 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:40:23.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:40:23.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:40:23.098 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:40:23.098 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 2020-03-11T22:40:23.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672974 -lt 2020-03-11T22:40:23.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672974: unary operator expected 2020-03-11T22:40:23.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:40:23.439 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-150323855363 2020-03-11T22:40:23.439 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:40:23.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:40:23.440 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-150323855363 2020-03-11T22:40:23.440 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:40:23.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=150323855363 2020-03-11T22:40:23.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 150323855363' 2020-03-11T22:40:23.441 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:40:23.441 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 150323855363 2020-03-11T22:40:23.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855363 -lt 150323855363 2020-03-11T22:40:23.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:40:23.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:23.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:24.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:40:24.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:24.243 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:24.243 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:24.244 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:24.245 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:24.245 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:24.245 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:24.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:24.574 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:24.574 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:24.575 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:25.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:40:25.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:40:25.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:40:25.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:40:25.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:25.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:40:25.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:25.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:25.116 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:25.117 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:25.117 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:25.117 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:25.117 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:25.117 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:25.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:25.447 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:25.447 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:25.447 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:25.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:40:25.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:25.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:25.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:25.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:25.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:25.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:25.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:25.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:26.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:26.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:26.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:40:26.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:26.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:40:26.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:26.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:26.535 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:26.535 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:26.535 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:26.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:26.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:26.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:26.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:26.865 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:26.865 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:26.865 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:27.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:40:27.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:27.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:27.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:27.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:27.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:27.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:27.310 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:27.311 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:27.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:27.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:27.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:40:27.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:27.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:40:28.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:28.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:28.157 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:28.157 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:28.157 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:28.157 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:28.158 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:28.158 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:28.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:28.488 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:28.488 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:28.488 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:28.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:40:28.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:28.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:28.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:28.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:28.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:28.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:28.931 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:28.931 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:29.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:29.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:29.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:40:29.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:29.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:40:30.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:30.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:30.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:30.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:30.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:30.185 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:30.185 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:30.185 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:30.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:30.529 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:30.529 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:30.530 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:30.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:40:30.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:30.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:30.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:30.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:30.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:30.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:30.974 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:30.974 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:31.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:31.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:31.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:40:31.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:31.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:40:33.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:33.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:33.022 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:33.022 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:33.022 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:33.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:33.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:33.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:33.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:33.365 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:33.365 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:33.365 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:33.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:40:33.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:33.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:33.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:33.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:33.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:33.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:33.813 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:33.813 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:34.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:34.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:34.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:40:34.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:34.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:40:37.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:37.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:37.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:37.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:37.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:37.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:37.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:37.461 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:37.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:37.802 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:37.802 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:37.802 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:38.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:40:38.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:38.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:38.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:38.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:38.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:38.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:38.246 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:38.246 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:38.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:38.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:38.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:40:38.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:38.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:40:45.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:45.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:45.116 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:45.116 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:45.116 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:45.116 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:45.116 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:45.116 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:45.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:45.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:45.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:45.461 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:45.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:40:45.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:45.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:45.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:45.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:45.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:45.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:45.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:45.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:46.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:46.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:46.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:40:46.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:40:46.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:40:53.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:40:53.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:53.688 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:53.688 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:53.688 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:53.688 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:53.688 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:53.689 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:54.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:54.030 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:54.031 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:54.031 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:54.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:40:54.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:40:54.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:40:54.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:40:54.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:40:54.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:40:54.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:40:54.471 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:40:54.472 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:40:54.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:40:54.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:40:54.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:40:54.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:40:55.343 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:40:55.344 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_fingerprint": "19898337-fb00-40eb-b748-56bfa6c9fb99", 2020-03-11T22:40:55.344 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:40:55.344 INFO:tasks.workunit.client.0.smithi145.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:40:55.344 INFO:tasks.workunit.client.0.smithi145.stdout: "timestamp": "2020-03-11T22:40:55.340569+0000", 2020-03-11T22:40:55.344 INFO:tasks.workunit.client.0.smithi145.stdout: "tag": "", 2020-03-11T22:40:55.345 INFO:tasks.workunit.client.0.smithi145.stdout: "health": { 2020-03-11T22:40:55.345 INFO:tasks.workunit.client.0.smithi145.stdout: "status": "HEALTH_WARN", 2020-03-11T22:40:55.345 INFO:tasks.workunit.client.0.smithi145.stdout: "checks": { 2020-03-11T22:40:55.345 INFO:tasks.workunit.client.0.smithi145.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:40:55.345 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:40:55.345 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2020-03-11T22:40:55.346 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:40:55.346 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 17 2020-03-11T22:40:55.346 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.346 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [], 2020-03-11T22:40:55.346 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2020-03-11T22:40:55.346 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.347 INFO:tasks.workunit.client.0.smithi145.stdout: "OSD_DOWN": { 2020-03-11T22:40:55.347 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:40:55.347 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2020-03-11T22:40:55.347 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "1 osds down", 2020-03-11T22:40:55.347 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 1 2020-03-11T22:40:55.347 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.347 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2020-03-11T22:40:55.348 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.348 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "osd.1 (root=default,host=smithi145) is down" 2020-03-11T22:40:55.348 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:40:55.348 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:40:55.348 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2020-03-11T22:40:55.348 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.348 INFO:tasks.workunit.client.0.smithi145.stdout: "PG_DEGRADED": { 2020-03-11T22:40:55.349 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:40:55.349 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2020-03-11T22:40:55.349 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2020-03-11T22:40:55.349 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 1 2020-03-11T22:40:55.349 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.349 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2020-03-11T22:40:55.350 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.350 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:40:55.350 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:40:55.350 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:40:55.350 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2020-03-11T22:40:55.350 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.351 INFO:tasks.workunit.client.0.smithi145.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:40:55.351 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:40:55.351 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2020-03-11T22:40:55.351 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:40:55.351 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 1 2020-03-11T22:40:55.351 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.352 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2020-03-11T22:40:55.352 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.352 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:40:55.352 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.352 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.352 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:40:55.352 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:40:55.352 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:40:55.353 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2020-03-11T22:40:55.353 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:40:55.353 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.353 INFO:tasks.workunit.client.0.smithi145.stdout: "mutes": [] 2020-03-11T22:40:55.353 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.353 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap_first_committed": 1, 2020-03-11T22:40:55.354 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap_last_committed": 1, 2020-03-11T22:40:55.354 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap": { 2020-03-11T22:40:55.354 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 1, 2020-03-11T22:40:55.354 INFO:tasks.workunit.client.0.smithi145.stdout: "fsid": "e665ac71-d8d3-412f-b706-8f1d72482f80", 2020-03-11T22:40:55.354 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2020-03-11T22:39:17.502728Z", 2020-03-11T22:40:55.354 INFO:tasks.workunit.client.0.smithi145.stdout: "created": "2020-03-11T22:39:17.502728Z", 2020-03-11T22:40:55.355 INFO:tasks.workunit.client.0.smithi145.stdout: "min_mon_release": 15, 2020-03-11T22:40:55.355 INFO:tasks.workunit.client.0.smithi145.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:40:55.355 INFO:tasks.workunit.client.0.smithi145.stdout: "features": { 2020-03-11T22:40:55.355 INFO:tasks.workunit.client.0.smithi145.stdout: "persistent": [ 2020-03-11T22:40:55.355 INFO:tasks.workunit.client.0.smithi145.stdout: "kraken", 2020-03-11T22:40:55.355 INFO:tasks.workunit.client.0.smithi145.stdout: "luminous", 2020-03-11T22:40:55.355 INFO:tasks.workunit.client.0.smithi145.stdout: "mimic", 2020-03-11T22:40:55.356 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap-prune", 2020-03-11T22:40:55.356 INFO:tasks.workunit.client.0.smithi145.stdout: "nautilus", 2020-03-11T22:40:55.356 INFO:tasks.workunit.client.0.smithi145.stdout: "octopus" 2020-03-11T22:40:55.356 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:40:55.356 INFO:tasks.workunit.client.0.smithi145.stdout: "optional": [] 2020-03-11T22:40:55.356 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.357 INFO:tasks.workunit.client.0.smithi145.stdout: "mons": [ 2020-03-11T22:40:55.357 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.357 INFO:tasks.workunit.client.0.smithi145.stdout: "rank": 0, 2020-03-11T22:40:55.357 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "a", 2020-03-11T22:40:55.357 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2020-03-11T22:40:55.357 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:40:55.357 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.358 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:40:55.358 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:40:55.358 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 0 2020-03-11T22:40:55.358 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:40:55.358 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:40:55.358 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.359 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:40:55.359 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:40:55.359 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 0, 2020-03-11T22:40:55.359 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 0 2020-03-11T22:40:55.359 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:40:55.359 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:40:55.359 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.360 INFO:tasks.workunit.client.0.smithi145.stdout: "quorum": [ 2020-03-11T22:40:55.360 INFO:tasks.workunit.client.0.smithi145.stdout: 0 2020-03-11T22:40:55.360 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:40:55.360 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap": { 2020-03-11T22:40:55.360 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 40, 2020-03-11T22:40:55.360 INFO:tasks.workunit.client.0.smithi145.stdout: "fsid": "e665ac71-d8d3-412f-b706-8f1d72482f80", 2020-03-11T22:40:55.360 INFO:tasks.workunit.client.0.smithi145.stdout: "created": "2020-03-11T22:39:18.079691+0000", 2020-03-11T22:40:55.361 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2020-03-11T22:40:19.317859+0000", 2020-03-11T22:40:55.361 INFO:tasks.workunit.client.0.smithi145.stdout: "last_up_change": "2020-03-11T22:40:17.555998+0000", 2020-03-11T22:40:55.361 INFO:tasks.workunit.client.0.smithi145.stdout: "last_in_change": "2020-03-11T22:40:19.021132+0000", 2020-03-11T22:40:55.361 INFO:tasks.workunit.client.0.smithi145.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:40:55.361 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_num": 5800384, 2020-03-11T22:40:55.361 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_set": [ 2020-03-11T22:40:55.362 INFO:tasks.workunit.client.0.smithi145.stdout: "nodown", 2020-03-11T22:40:55.362 INFO:tasks.workunit.client.0.smithi145.stdout: "noin", 2020-03-11T22:40:55.362 INFO:tasks.workunit.client.0.smithi145.stdout: "noout", 2020-03-11T22:40:55.362 INFO:tasks.workunit.client.0.smithi145.stdout: "pglog_hardlimit", 2020-03-11T22:40:55.362 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snapdirs", 2020-03-11T22:40:55.362 INFO:tasks.workunit.client.0.smithi145.stdout: "recovery_deletes", 2020-03-11T22:40:55.362 INFO:tasks.workunit.client.0.smithi145.stdout: "sortbitwise" 2020-03-11T22:40:55.363 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:40:55.363 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_version": 7, 2020-03-11T22:40:55.363 INFO:tasks.workunit.client.0.smithi145.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:40:55.363 INFO:tasks.workunit.client.0.smithi145.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:40:55.363 INFO:tasks.workunit.client.0.smithi145.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:40:55.363 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_snapshot": "", 2020-03-11T22:40:55.363 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_max": 1, 2020-03-11T22:40:55.364 INFO:tasks.workunit.client.0.smithi145.stdout: "max_osd": 3, 2020-03-11T22:40:55.364 INFO:tasks.workunit.client.0.smithi145.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:40:55.364 INFO:tasks.workunit.client.0.smithi145.stdout: "min_compat_client": "jewel", 2020-03-11T22:40:55.364 INFO:tasks.workunit.client.0.smithi145.stdout: "require_osd_release": "octopus", 2020-03-11T22:40:55.364 INFO:tasks.workunit.client.0.smithi145.stdout: "pools": [ 2020-03-11T22:40:55.364 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.364 INFO:tasks.workunit.client.0.smithi145.stdout: "pool": 1, 2020-03-11T22:40:55.365 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_name": "test", 2020-03-11T22:40:55.365 INFO:tasks.workunit.client.0.smithi145.stdout: "create_time": "2020-03-11T22:39:39.166414+0000", 2020-03-11T22:40:55.365 INFO:tasks.workunit.client.0.smithi145.stdout: "flags": 1, 2020-03-11T22:40:55.365 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_names": "hashpspool", 2020-03-11T22:40:55.365 INFO:tasks.workunit.client.0.smithi145.stdout: "type": 1, 2020-03-11T22:40:55.365 INFO:tasks.workunit.client.0.smithi145.stdout: "size": 3, 2020-03-11T22:40:55.366 INFO:tasks.workunit.client.0.smithi145.stdout: "min_size": 2, 2020-03-11T22:40:55.366 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_rule": 0, 2020-03-11T22:40:55.366 INFO:tasks.workunit.client.0.smithi145.stdout: "object_hash": 2, 2020-03-11T22:40:55.366 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_autoscale_mode": "off", 2020-03-11T22:40:55.366 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num": 1, 2020-03-11T22:40:55.366 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_placement_num": 1, 2020-03-11T22:40:55.366 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_placement_num_target": 1, 2020-03-11T22:40:55.367 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num_target": 1, 2020-03-11T22:40:55.367 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num_pending": 1, 2020-03-11T22:40:55.367 INFO:tasks.workunit.client.0.smithi145.stdout: "last_pg_merge_meta": { 2020-03-11T22:40:55.367 INFO:tasks.workunit.client.0.smithi145.stdout: "source_pgid": "0.0", 2020-03-11T22:40:55.367 INFO:tasks.workunit.client.0.smithi145.stdout: "ready_epoch": 0, 2020-03-11T22:40:55.367 INFO:tasks.workunit.client.0.smithi145.stdout: "last_epoch_started": 0, 2020-03-11T22:40:55.367 INFO:tasks.workunit.client.0.smithi145.stdout: "last_epoch_clean": 0, 2020-03-11T22:40:55.368 INFO:tasks.workunit.client.0.smithi145.stdout: "source_version": "0'0", 2020-03-11T22:40:55.368 INFO:tasks.workunit.client.0.smithi145.stdout: "target_version": "0'0" 2020-03-11T22:40:55.368 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.368 INFO:tasks.workunit.client.0.smithi145.stdout: "last_change": "28", 2020-03-11T22:40:55.368 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend": "0", 2020-03-11T22:40:55.368 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:40:55.369 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend_preluminous": "0", 2020-03-11T22:40:55.369 INFO:tasks.workunit.client.0.smithi145.stdout: "auid": 0, 2020-03-11T22:40:55.369 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_mode": "selfmanaged", 2020-03-11T22:40:55.369 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_seq": 0, 2020-03-11T22:40:55.369 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_epoch": 0, 2020-03-11T22:40:55.369 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_snaps": [], 2020-03-11T22:40:55.370 INFO:tasks.workunit.client.0.smithi145.stdout: "removed_snaps": "[]", 2020-03-11T22:40:55.370 INFO:tasks.workunit.client.0.smithi145.stdout: "quota_max_bytes": 0, 2020-03-11T22:40:55.370 INFO:tasks.workunit.client.0.smithi145.stdout: "quota_max_objects": 0, 2020-03-11T22:40:55.370 INFO:tasks.workunit.client.0.smithi145.stdout: "tiers": [], 2020-03-11T22:40:55.370 INFO:tasks.workunit.client.0.smithi145.stdout: "tier_of": -1, 2020-03-11T22:40:55.370 INFO:tasks.workunit.client.0.smithi145.stdout: "read_tier": -1, 2020-03-11T22:40:55.371 INFO:tasks.workunit.client.0.smithi145.stdout: "write_tier": -1, 2020-03-11T22:40:55.371 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_mode": "none", 2020-03-11T22:40:55.371 INFO:tasks.workunit.client.0.smithi145.stdout: "target_max_bytes": 0, 2020-03-11T22:40:55.371 INFO:tasks.workunit.client.0.smithi145.stdout: "target_max_objects": 0, 2020-03-11T22:40:55.371 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:40:55.371 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:40:55.372 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:40:55.372 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_min_flush_age": 0, 2020-03-11T22:40:55.372 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_min_evict_age": 0, 2020-03-11T22:40:55.372 INFO:tasks.workunit.client.0.smithi145.stdout: "erasure_code_profile": "", 2020-03-11T22:40:55.372 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_params": { 2020-03-11T22:40:55.372 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "none" 2020-03-11T22:40:55.372 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.373 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_period": 0, 2020-03-11T22:40:55.373 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_count": 0, 2020-03-11T22:40:55.373 INFO:tasks.workunit.client.0.smithi145.stdout: "use_gmt_hitset": true, 2020-03-11T22:40:55.373 INFO:tasks.workunit.client.0.smithi145.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:40:55.373 INFO:tasks.workunit.client.0.smithi145.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:40:55.373 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:40:55.374 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:40:55.374 INFO:tasks.workunit.client.0.smithi145.stdout: "grade_table": [], 2020-03-11T22:40:55.374 INFO:tasks.workunit.client.0.smithi145.stdout: "stripe_width": 0, 2020-03-11T22:40:55.374 INFO:tasks.workunit.client.0.smithi145.stdout: "expected_num_objects": 0, 2020-03-11T22:40:55.374 INFO:tasks.workunit.client.0.smithi145.stdout: "fast_read": false, 2020-03-11T22:40:55.374 INFO:tasks.workunit.client.0.smithi145.stdout: "options": {}, 2020-03-11T22:40:55.374 INFO:tasks.workunit.client.0.smithi145.stdout: "application_metadata": {} 2020-03-11T22:40:55.375 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:40:55.375 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:40:55.375 INFO:tasks.workunit.client.0.smithi145.stdout: "osds": [ 2020-03-11T22:40:55.375 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.375 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2020-03-11T22:40:55.375 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "6548d5c2-3fe9-45a6-8404-614e80715021", 2020-03-11T22:40:55.375 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 1, 2020-03-11T22:40:55.376 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2020-03-11T22:40:55.376 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2020-03-11T22:40:55.376 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2020-03-11T22:40:55.376 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 5, 2020-03-11T22:40:55.376 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 28, 2020-03-11T22:40:55.376 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 34, 2020-03-11T22:40:55.377 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 39, 2020-03-11T22:40:55.377 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 29, 2020-03-11T22:40:55.377 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2020-03-11T22:40:55.377 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2020-03-11T22:40:55.377 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:40:55.377 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.378 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:40:55.378 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:40:55.378 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 30195 2020-03-11T22:40:55.378 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.378 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.378 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:40:55.379 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:40:55.379 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 30195 2020-03-11T22:40:55.379 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:40:55.379 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:40:55.379 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.379 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2020-03-11T22:40:55.379 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:40:55.380 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.380 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:40:55.380 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:40:55.380 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 30195 2020-03-11T22:40:55.380 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.380 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.380 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:40:55.381 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:40:55.381 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 30195 2020-03-11T22:40:55.381 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:40:55.381 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:40:55.381 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.381 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2020-03-11T22:40:55.381 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:40:55.382 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.382 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:40:55.382 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:40:55.382 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 30195 2020-03-11T22:40:55.382 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.382 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.383 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:40:55.383 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:40:55.383 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 30195 2020-03-11T22:40:55.383 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:40:55.383 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:40:55.383 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.384 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2020-03-11T22:40:55.384 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:40:55.384 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.384 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:40:55.384 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:40:55.384 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 30195 2020-03-11T22:40:55.384 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.385 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.385 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:40:55.385 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:40:55.385 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 30195 2020-03-11T22:40:55.385 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:40:55.385 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:40:55.385 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.386 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6803/30195", 2020-03-11T22:40:55.386 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6805/30195", 2020-03-11T22:40:55.386 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6809/30195", 2020-03-11T22:40:55.386 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6807/30195", 2020-03-11T22:40:55.386 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2020-03-11T22:40:55.386 INFO:tasks.workunit.client.0.smithi145.stdout: "exists", 2020-03-11T22:40:55.387 INFO:tasks.workunit.client.0.smithi145.stdout: "up" 2020-03-11T22:40:55.387 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:40:55.387 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.387 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.387 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2020-03-11T22:40:55.387 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "fc159c74-e2d8-44ac-b32a-c54afc816878", 2020-03-11T22:40:55.388 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 0, 2020-03-11T22:40:55.388 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2020-03-11T22:40:55.388 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2020-03-11T22:40:55.388 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2020-03-11T22:40:55.388 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 0, 2020-03-11T22:40:55.388 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 0, 2020-03-11T22:40:55.388 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 10, 2020-03-11T22:40:55.388 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 22, 2020-03-11T22:40:55.389 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 30, 2020-03-11T22:40:55.389 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2020-03-11T22:40:55.389 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2020-03-11T22:40:55.389 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:40:55.389 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.390 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:40:55.390 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:40:55.390 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 25828 2020-03-11T22:40:55.390 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.390 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.390 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:40:55.391 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:40:55.391 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 25828 2020-03-11T22:40:55.391 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:40:55.391 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:40:55.391 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.391 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2020-03-11T22:40:55.391 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:40:55.392 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.392 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:40:55.392 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:40:55.392 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 25828 2020-03-11T22:40:55.392 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.392 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.392 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:40:55.393 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:40:55.393 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 25828 2020-03-11T22:40:55.393 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:40:55.393 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:40:55.393 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.393 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2020-03-11T22:40:55.394 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:40:55.394 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.394 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:40:55.394 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:40:55.394 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 25828 2020-03-11T22:40:55.394 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.395 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.395 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:40:55.395 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:40:55.396 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 25828 2020-03-11T22:40:55.397 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:40:55.397 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:40:55.397 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.397 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2020-03-11T22:40:55.397 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:40:55.397 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.398 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:40:55.398 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:40:55.398 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 25828 2020-03-11T22:40:55.398 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.398 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.399 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:40:55.399 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:40:55.399 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 25828 2020-03-11T22:40:55.399 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:40:55.399 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:40:55.399 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.400 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6811/25828", 2020-03-11T22:40:55.400 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6813/25828", 2020-03-11T22:40:55.400 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6817/25828", 2020-03-11T22:40:55.400 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6815/25828", 2020-03-11T22:40:55.400 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2020-03-11T22:40:55.400 INFO:tasks.workunit.client.0.smithi145.stdout: "exists" 2020-03-11T22:40:55.401 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:40:55.401 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.401 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.401 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2020-03-11T22:40:55.401 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "cc077be5-2bbd-488f-9867-ebfa384fe3f6", 2020-03-11T22:40:55.401 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 1, 2020-03-11T22:40:55.401 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2020-03-11T22:40:55.402 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2020-03-11T22:40:55.402 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2020-03-11T22:40:55.402 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 15, 2020-03-11T22:40:55.402 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 28, 2020-03-11T22:40:55.402 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 35, 2020-03-11T22:40:55.402 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 0, 2020-03-11T22:40:55.403 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 31, 2020-03-11T22:40:55.403 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2020-03-11T22:40:55.403 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2020-03-11T22:40:55.403 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:40:55.403 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.403 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:40:55.404 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:40:55.404 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 30389 2020-03-11T22:40:55.404 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.404 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.404 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:40:55.404 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:40:55.404 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 30389 2020-03-11T22:40:55.405 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:40:55.405 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:40:55.405 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.405 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2020-03-11T22:40:55.405 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:40:55.405 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.405 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:40:55.406 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:40:55.406 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 30389 2020-03-11T22:40:55.406 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.406 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.406 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:40:55.406 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:40:55.406 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 30389 2020-03-11T22:40:55.407 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:40:55.407 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:40:55.407 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.407 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2020-03-11T22:40:55.407 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:40:55.407 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.408 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:40:55.408 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:40:55.408 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 30389 2020-03-11T22:40:55.408 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.408 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.408 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:40:55.408 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:40:55.409 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 30389 2020-03-11T22:40:55.409 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:40:55.409 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:40:55.409 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.409 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2020-03-11T22:40:55.409 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:40:55.409 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.410 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:40:55.410 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:40:55.410 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 30389 2020-03-11T22:40:55.410 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.410 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.410 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:40:55.410 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:40:55.411 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 30389 2020-03-11T22:40:55.411 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:40:55.411 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:40:55.411 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.411 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6811/30389", 2020-03-11T22:40:55.411 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6813/30389", 2020-03-11T22:40:55.411 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6817/30389", 2020-03-11T22:40:55.412 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6815/30389", 2020-03-11T22:40:55.412 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2020-03-11T22:40:55.412 INFO:tasks.workunit.client.0.smithi145.stdout: "exists", 2020-03-11T22:40:55.412 INFO:tasks.workunit.client.0.smithi145.stdout: "up" 2020-03-11T22:40:55.412 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:40:55.412 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:40:55.413 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:40:55.413 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_xinfo": [ 2020-03-11T22:40:55.413 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.413 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2020-03-11T22:40:55.413 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "2020-03-11T22:40:11.564170+0000", 2020-03-11T22:40:55.413 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2020-03-11T22:40:55.413 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2020-03-11T22:40:55.414 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138292836696063, 2020-03-11T22:40:55.414 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2020-03-11T22:40:55.414 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2020-03-11T22:39:24.632044+0000", 2020-03-11T22:40:55.414 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2020-03-11T22:40:55.414 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.414 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.414 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2020-03-11T22:40:55.415 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "2020-03-11T22:40:12.041763+0000", 2020-03-11T22:40:55.415 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2020-03-11T22:40:55.415 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2020-03-11T22:40:55.415 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138292836696063, 2020-03-11T22:40:55.415 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2020-03-11T22:40:55.415 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2020-03-11T22:39:30.875922+0000", 2020-03-11T22:40:55.415 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2020-03-11T22:40:55.416 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.416 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.416 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2020-03-11T22:40:55.416 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "2020-03-11T22:40:12.508032+0000", 2020-03-11T22:40:55.416 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2020-03-11T22:40:55.416 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2020-03-11T22:40:55.417 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138292836696063, 2020-03-11T22:40:55.417 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2020-03-11T22:40:55.417 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2020-03-11T22:39:37.114069+0000", 2020-03-11T22:40:55.417 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2020-03-11T22:40:55.417 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:40:55.417 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:40:55.418 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_upmap": [], 2020-03-11T22:40:55.418 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_upmap_items": [], 2020-03-11T22:40:55.418 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_temp": [], 2020-03-11T22:40:55.418 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_temp": [], 2020-03-11T22:40:55.418 INFO:tasks.workunit.client.0.smithi145.stdout: "blacklist": {}, 2020-03-11T22:40:55.418 INFO:tasks.workunit.client.0.smithi145.stdout: "erasure_code_profiles": { 2020-03-11T22:40:55.418 INFO:tasks.workunit.client.0.smithi145.stdout: "default": { 2020-03-11T22:40:55.419 INFO:tasks.workunit.client.0.smithi145.stdout: "k": "2", 2020-03-11T22:40:55.419 INFO:tasks.workunit.client.0.smithi145.stdout: "m": "2", 2020-03-11T22:40:55.419 INFO:tasks.workunit.client.0.smithi145.stdout: "plugin": "jerasure", 2020-03-11T22:40:55.419 INFO:tasks.workunit.client.0.smithi145.stdout: "technique": "reed_sol_van" 2020-03-11T22:40:55.419 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:40:55.419 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.420 INFO:tasks.workunit.client.0.smithi145.stdout: "removed_snaps_queue": [], 2020-03-11T22:40:55.420 INFO:tasks.workunit.client.0.smithi145.stdout: "new_removed_snaps": [], 2020-03-11T22:40:55.420 INFO:tasks.workunit.client.0.smithi145.stdout: "new_purged_snaps": [], 2020-03-11T22:40:55.420 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_node_flags": {}, 2020-03-11T22:40:55.420 INFO:tasks.workunit.client.0.smithi145.stdout: "device_class_flags": {} 2020-03-11T22:40:55.420 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.420 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_metadata": [ 2020-03-11T22:40:55.421 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.421 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2020-03-11T22:40:55.421 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2020-03-11T22:40:55.421 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6804/30195,v1:127.0.0.1:6805/30195]", 2020-03-11T22:40:55.421 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2020-03-11T22:40:55.421 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2020-03-11T22:40:55.422 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:40:55.422 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:40:55.422 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:40:55.422 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:40:55.422 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:40:55.422 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:40:55.423 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:40:55.423 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:40:55.423 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:40:55.423 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:40:55.423 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:40:55.423 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:40:55.424 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "octopus", 2020-03-11T22:40:55.424 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:40:55.424 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:40:55.424 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:40:55.424 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2020-03-11T22:40:55.424 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XLDP", 2020-03-11T22:40:55.425 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:40:55.425 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2020-03-11T22:40:55.425 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2020-03-11T22:40:55.425 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:40:55.425 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2020-03-11T22:40:55.425 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6802/30195,v1:127.0.0.1:6803/30195]", 2020-03-11T22:40:55.425 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2020-03-11T22:40:55.426 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/30195,v1:127.0.0.1:6809/30195]", 2020-03-11T22:40:55.426 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/30195,v1:127.0.0.1:6807/30195]", 2020-03-11T22:40:55.426 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2020-03-11T22:40:55.426 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2020-03-11T22:40:55.426 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:40:55.426 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:40:55.426 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2020-03-11T22:40:55.427 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32655820", 2020-03-11T22:40:55.427 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:40:55.427 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:40:55.427 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2020-03-11T22:40:55.427 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:40:55.427 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:40:55.427 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2020-03-11T22:40:55.428 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.428 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.428 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2020-03-11T22:40:55.428 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2020-03-11T22:40:55.428 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6812/25828,v1:127.0.0.1:6813/25828]", 2020-03-11T22:40:55.428 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2020-03-11T22:40:55.428 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2020-03-11T22:40:55.429 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:40:55.429 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:40:55.429 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:40:55.429 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:40:55.429 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:40:55.429 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:40:55.430 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:40:55.430 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:40:55.430 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:40:55.430 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:40:55.430 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:40:55.430 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:40:55.431 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "octopus", 2020-03-11T22:40:55.431 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:40:55.431 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:40:55.431 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:40:55.431 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2020-03-11T22:40:55.431 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XLDP", 2020-03-11T22:40:55.432 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:40:55.432 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2020-03-11T22:40:55.432 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2020-03-11T22:40:55.432 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:40:55.432 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2020-03-11T22:40:55.432 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6810/25828,v1:127.0.0.1:6811/25828]", 2020-03-11T22:40:55.433 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2020-03-11T22:40:55.433 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/25828,v1:127.0.0.1:6817/25828]", 2020-03-11T22:40:55.433 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/25828,v1:127.0.0.1:6815/25828]", 2020-03-11T22:40:55.433 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2020-03-11T22:40:55.433 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2020-03-11T22:40:55.433 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:40:55.434 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:40:55.434 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2020-03-11T22:40:55.434 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32655820", 2020-03-11T22:40:55.434 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:40:55.434 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:40:55.434 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2020-03-11T22:40:55.435 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:40:55.435 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:40:55.435 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2020-03-11T22:40:55.435 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.435 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.435 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2020-03-11T22:40:55.436 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2020-03-11T22:40:55.436 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6812/30389,v1:127.0.0.1:6813/30389]", 2020-03-11T22:40:55.436 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2020-03-11T22:40:55.436 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2020-03-11T22:40:55.436 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:40:55.436 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:40:55.436 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:40:55.437 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:40:55.437 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:40:55.437 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:40:55.437 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:40:55.437 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:40:55.438 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:40:55.438 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:40:55.438 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:40:55.438 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:40:55.438 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "octopus", 2020-03-11T22:40:55.438 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:40:55.438 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:40:55.439 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:40:55.439 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2020-03-11T22:40:55.439 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XLDP", 2020-03-11T22:40:55.439 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:40:55.439 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2020-03-11T22:40:55.439 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2020-03-11T22:40:55.439 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:40:55.440 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2020-03-11T22:40:55.440 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6810/30389,v1:127.0.0.1:6811/30389]", 2020-03-11T22:40:55.440 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2020-03-11T22:40:55.440 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/30389,v1:127.0.0.1:6817/30389]", 2020-03-11T22:40:55.440 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/30389,v1:127.0.0.1:6815/30389]", 2020-03-11T22:40:55.440 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2020-03-11T22:40:55.440 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2020-03-11T22:40:55.441 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:40:55.441 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:40:55.441 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2020-03-11T22:40:55.441 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32655820", 2020-03-11T22:40:55.441 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:40:55.441 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:40:55.442 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2020-03-11T22:40:55.442 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:40:55.442 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:40:55.442 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2020-03-11T22:40:55.442 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:40:55.442 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:40:55.443 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap_first_committed": 1, 2020-03-11T22:40:55.443 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap_last_committed": 40, 2020-03-11T22:40:55.443 INFO:tasks.workunit.client.0.smithi145.stdout: "crushmap": { 2020-03-11T22:40:55.443 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": [ 2020-03-11T22:40:55.443 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.443 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2020-03-11T22:40:55.443 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.0", 2020-03-11T22:40:55.444 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2020-03-11T22:40:55.444 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.444 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.444 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2020-03-11T22:40:55.444 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.1", 2020-03-11T22:40:55.444 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2020-03-11T22:40:55.444 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.445 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.445 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2020-03-11T22:40:55.445 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.2", 2020-03-11T22:40:55.445 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2020-03-11T22:40:55.445 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:40:55.445 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:40:55.445 INFO:tasks.workunit.client.0.smithi145.stdout: "types": [ 2020-03-11T22:40:55.446 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.446 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 0, 2020-03-11T22:40:55.446 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd" 2020-03-11T22:40:55.446 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.446 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.446 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2020-03-11T22:40:55.447 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "host" 2020-03-11T22:40:55.447 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.447 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.447 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 2, 2020-03-11T22:40:55.447 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "chassis" 2020-03-11T22:40:55.447 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.448 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.448 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 3, 2020-03-11T22:40:55.448 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "rack" 2020-03-11T22:40:55.448 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.448 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.448 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 4, 2020-03-11T22:40:55.449 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "row" 2020-03-11T22:40:55.449 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.449 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.449 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 5, 2020-03-11T22:40:55.449 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "pdu" 2020-03-11T22:40:55.449 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.450 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.450 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 6, 2020-03-11T22:40:55.450 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "pod" 2020-03-11T22:40:55.450 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.450 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.450 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 7, 2020-03-11T22:40:55.450 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "room" 2020-03-11T22:40:55.451 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.451 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.451 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 8, 2020-03-11T22:40:55.451 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "datacenter" 2020-03-11T22:40:55.451 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.451 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.452 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 9, 2020-03-11T22:40:55.452 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "zone" 2020-03-11T22:40:55.452 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.452 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.452 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 10, 2020-03-11T22:40:55.452 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "region" 2020-03-11T22:40:55.453 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.453 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.453 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2020-03-11T22:40:55.453 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "root" 2020-03-11T22:40:55.453 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:40:55.453 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:40:55.453 INFO:tasks.workunit.client.0.smithi145.stdout: "buckets": [ 2020-03-11T22:40:55.454 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.454 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -1, 2020-03-11T22:40:55.454 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "default", 2020-03-11T22:40:55.454 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2020-03-11T22:40:55.454 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "root", 2020-03-11T22:40:55.454 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2020-03-11T22:40:55.455 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2020-03-11T22:40:55.455 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2020-03-11T22:40:55.455 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:40:55.455 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.455 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -3, 2020-03-11T22:40:55.455 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2020-03-11T22:40:55.456 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2020-03-11T22:40:55.456 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:40:55.456 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:40:55.456 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.456 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.456 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -2, 2020-03-11T22:40:55.457 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "default~hdd", 2020-03-11T22:40:55.457 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2020-03-11T22:40:55.457 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "root", 2020-03-11T22:40:55.457 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2020-03-11T22:40:55.457 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2020-03-11T22:40:55.457 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2020-03-11T22:40:55.458 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:40:55.458 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.458 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -4, 2020-03-11T22:40:55.458 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2020-03-11T22:40:55.458 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2020-03-11T22:40:55.458 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:40:55.458 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:40:55.459 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.459 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.459 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -3, 2020-03-11T22:40:55.459 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "smithi145", 2020-03-11T22:40:55.459 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2020-03-11T22:40:55.459 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "host", 2020-03-11T22:40:55.459 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2020-03-11T22:40:55.460 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2020-03-11T22:40:55.460 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2020-03-11T22:40:55.460 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:40:55.460 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.460 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2020-03-11T22:40:55.461 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2020-03-11T22:40:55.461 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2020-03-11T22:40:55.461 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.461 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.461 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2020-03-11T22:40:55.461 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2020-03-11T22:40:55.461 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 1 2020-03-11T22:40:55.461 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.462 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.462 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2020-03-11T22:40:55.462 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2020-03-11T22:40:55.462 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 2 2020-03-11T22:40:55.462 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:40:55.462 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:40:55.462 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.463 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.463 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -4, 2020-03-11T22:40:55.463 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "smithi145~hdd", 2020-03-11T22:40:55.463 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2020-03-11T22:40:55.463 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "host", 2020-03-11T22:40:55.463 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2020-03-11T22:40:55.464 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2020-03-11T22:40:55.464 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2020-03-11T22:40:55.465 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:40:55.465 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.465 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2020-03-11T22:40:55.465 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2020-03-11T22:40:55.465 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2020-03-11T22:40:55.465 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.465 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.466 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2020-03-11T22:40:55.466 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2020-03-11T22:40:55.466 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 1 2020-03-11T22:40:55.466 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.466 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.466 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2020-03-11T22:40:55.466 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2020-03-11T22:40:55.467 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 2 2020-03-11T22:40:55.467 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:40:55.467 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:40:55.467 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:40:55.467 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:40:55.467 INFO:tasks.workunit.client.0.smithi145.stdout: "rules": [ 2020-03-11T22:40:55.468 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.468 INFO:tasks.workunit.client.0.smithi145.stdout: "rule_id": 0, 2020-03-11T22:40:55.468 INFO:tasks.workunit.client.0.smithi145.stdout: "rule_name": "replicated_rule", 2020-03-11T22:40:55.468 INFO:tasks.workunit.client.0.smithi145.stdout: "ruleset": 0, 2020-03-11T22:40:55.468 INFO:tasks.workunit.client.0.smithi145.stdout: "type": 1, 2020-03-11T22:40:55.468 INFO:tasks.workunit.client.0.smithi145.stdout: "min_size": 1, 2020-03-11T22:40:55.469 INFO:tasks.workunit.client.0.smithi145.stdout: "max_size": 10, 2020-03-11T22:40:55.469 INFO:tasks.workunit.client.0.smithi145.stdout: "steps": [ 2020-03-11T22:40:55.469 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.469 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "take", 2020-03-11T22:40:55.469 INFO:tasks.workunit.client.0.smithi145.stdout: "item": -1, 2020-03-11T22:40:55.469 INFO:tasks.workunit.client.0.smithi145.stdout: "item_name": "default" 2020-03-11T22:40:55.469 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.470 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.470 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "choose_firstn", 2020-03-11T22:40:55.470 INFO:tasks.workunit.client.0.smithi145.stdout: "num": 0, 2020-03-11T22:40:55.470 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "osd" 2020-03-11T22:40:55.470 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.470 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.470 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "emit" 2020-03-11T22:40:55.471 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:40:55.471 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:40:55.471 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:40:55.471 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:40:55.471 INFO:tasks.workunit.client.0.smithi145.stdout: "tunables": { 2020-03-11T22:40:55.472 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_local_tries": 0, 2020-03-11T22:40:55.472 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:40:55.472 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_total_tries": 50, 2020-03-11T22:40:55.472 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:40:55.472 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:40:55.472 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_stable": 1, 2020-03-11T22:40:55.472 INFO:tasks.workunit.client.0.smithi145.stdout: "straw_calc_version": 1, 2020-03-11T22:40:55.473 INFO:tasks.workunit.client.0.smithi145.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:40:55.473 INFO:tasks.workunit.client.0.smithi145.stdout: "profile": "jewel", 2020-03-11T22:40:55.473 INFO:tasks.workunit.client.0.smithi145.stdout: "optimal_tunables": 1, 2020-03-11T22:40:55.473 INFO:tasks.workunit.client.0.smithi145.stdout: "legacy_tunables": 0, 2020-03-11T22:40:55.473 INFO:tasks.workunit.client.0.smithi145.stdout: "minimum_required_version": "jewel", 2020-03-11T22:40:55.473 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables": 1, 2020-03-11T22:40:55.473 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables2": 1, 2020-03-11T22:40:55.474 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v2_rules": 0, 2020-03-11T22:40:55.474 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables3": 1, 2020-03-11T22:40:55.474 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v3_rules": 0, 2020-03-11T22:40:55.474 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v4_buckets": 1, 2020-03-11T22:40:55.474 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables5": 1, 2020-03-11T22:40:55.474 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v5_rules": 0 2020-03-11T22:40:55.474 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.475 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_args": {} 2020-03-11T22:40:55.475 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.475 INFO:tasks.workunit.client.0.smithi145.stdout: "fsmap": { 2020-03-11T22:40:55.475 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 1, 2020-03-11T22:40:55.475 INFO:tasks.workunit.client.0.smithi145.stdout: "default_fscid": -1, 2020-03-11T22:40:55.475 INFO:tasks.workunit.client.0.smithi145.stdout: "compat": { 2020-03-11T22:40:55.476 INFO:tasks.workunit.client.0.smithi145.stdout: "compat": {}, 2020-03-11T22:40:55.476 INFO:tasks.workunit.client.0.smithi145.stdout: "ro_compat": {}, 2020-03-11T22:40:55.476 INFO:tasks.workunit.client.0.smithi145.stdout: "incompat": { 2020-03-11T22:40:55.476 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_1": "base v0.20", 2020-03-11T22:40:55.476 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:40:55.476 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:40:55.476 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:40:55.477 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:40:55.477 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:40:55.477 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_8": "no anchor table", 2020-03-11T22:40:55.477 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_9": "file layout v2", 2020-03-11T22:40:55.477 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:40:55.477 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:40:55.478 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.478 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_flags": { 2020-03-11T22:40:55.478 INFO:tasks.workunit.client.0.smithi145.stdout: "enable_multiple": false, 2020-03-11T22:40:55.478 INFO:tasks.workunit.client.0.smithi145.stdout: "ever_enabled_multiple": false 2020-03-11T22:40:55.478 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.478 INFO:tasks.workunit.client.0.smithi145.stdout: "standbys": [], 2020-03-11T22:40:55.478 INFO:tasks.workunit.client.0.smithi145.stdout: "filesystems": [] 2020-03-11T22:40:55.479 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.479 INFO:tasks.workunit.client.0.smithi145.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:40:55.479 INFO:tasks.workunit.client.0.smithi145.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:40:55.479 INFO:tasks.workunit.client.0.smithi145.stdout: "auth": { 2020-03-11T22:40:55.479 INFO:tasks.workunit.client.0.smithi145.stdout: "first_committed": 1, 2020-03-11T22:40:55.479 INFO:tasks.workunit.client.0.smithi145.stdout: "last_committed": 5, 2020-03-11T22:40:55.479 INFO:tasks.workunit.client.0.smithi145.stdout: "num_secrets": 3 2020-03-11T22:40:55.480 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.480 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg": 1, 2020-03-11T22:40:55.480 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_active": 1, 2020-03-11T22:40:55.480 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_unknown": 0, 2020-03-11T22:40:55.480 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osd": 3, 2020-03-11T22:40:55.480 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_sum": { 2020-03-11T22:40:55.480 INFO:tasks.workunit.client.0.smithi145.stdout: "stat_sum": { 2020-03-11T22:40:55.481 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes": 48300, 2020-03-11T22:40:55.481 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects": 100, 2020-03-11T22:40:55.481 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_clones": 0, 2020-03-11T22:40:55.481 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_copies": 300, 2020-03-11T22:40:55.481 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:40:55.481 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing": 0, 2020-03-11T22:40:55.481 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_degraded": 100, 2020-03-11T22:40:55.482 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_misplaced": 0, 2020-03-11T22:40:55.482 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_unfound": 0, 2020-03-11T22:40:55.482 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_dirty": 100, 2020-03-11T22:40:55.482 INFO:tasks.workunit.client.0.smithi145.stdout: "num_whiteouts": 0, 2020-03-11T22:40:55.482 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read": 0, 2020-03-11T22:40:55.482 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read_kb": 0, 2020-03-11T22:40:55.483 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write": 100, 2020-03-11T22:40:55.483 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write_kb": 100, 2020-03-11T22:40:55.483 INFO:tasks.workunit.client.0.smithi145.stdout: "num_scrub_errors": 0, 2020-03-11T22:40:55.483 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:40:55.483 INFO:tasks.workunit.client.0.smithi145.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:40:55.483 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_recovered": 0, 2020-03-11T22:40:55.483 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_recovered": 0, 2020-03-11T22:40:55.484 INFO:tasks.workunit.client.0.smithi145.stdout: "num_keys_recovered": 0, 2020-03-11T22:40:55.484 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_omap": 0, 2020-03-11T22:40:55.484 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:40:55.484 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:40:55.484 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush": 0, 2020-03-11T22:40:55.484 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_kb": 0, 2020-03-11T22:40:55.484 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict": 0, 2020-03-11T22:40:55.485 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_kb": 0, 2020-03-11T22:40:55.485 INFO:tasks.workunit.client.0.smithi145.stdout: "num_promote": 0, 2020-03-11T22:40:55.485 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_high": 0, 2020-03-11T22:40:55.485 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_low": 0, 2020-03-11T22:40:55.485 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_some": 0, 2020-03-11T22:40:55.485 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_full": 0, 2020-03-11T22:40:55.486 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_pinned": 0, 2020-03-11T22:40:55.486 INFO:tasks.workunit.client.0.smithi145.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:40:55.486 INFO:tasks.workunit.client.0.smithi145.stdout: "num_large_omap_objects": 0, 2020-03-11T22:40:55.486 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_manifest": 0, 2020-03-11T22:40:55.486 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_bytes": 0, 2020-03-11T22:40:55.486 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_keys": 0, 2020-03-11T22:40:55.486 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_repaired": 0 2020-03-11T22:40:55.487 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.487 INFO:tasks.workunit.client.0.smithi145.stdout: "store_stats": { 2020-03-11T22:40:55.487 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 0, 2020-03-11T22:40:55.487 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 0, 2020-03-11T22:40:55.487 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2020-03-11T22:40:55.487 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 0, 2020-03-11T22:40:55.487 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 0, 2020-03-11T22:40:55.488 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2020-03-11T22:40:55.488 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2020-03-11T22:40:55.488 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2020-03-11T22:40:55.488 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 0, 2020-03-11T22:40:55.488 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 0 2020-03-11T22:40:55.488 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.488 INFO:tasks.workunit.client.0.smithi145.stdout: "log_size": 16, 2020-03-11T22:40:55.489 INFO:tasks.workunit.client.0.smithi145.stdout: "ondisk_log_size": 16, 2020-03-11T22:40:55.489 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 2, 2020-03-11T22:40:55.489 INFO:tasks.workunit.client.0.smithi145.stdout: "acting": 2, 2020-03-11T22:40:55.489 INFO:tasks.workunit.client.0.smithi145.stdout: "num_store_stats": 0 2020-03-11T22:40:55.489 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.489 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_sum": { 2020-03-11T22:40:55.490 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 0, 2020-03-11T22:40:55.490 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 0, 2020-03-11T22:40:55.490 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pgs": 3, 2020-03-11T22:40:55.490 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osds": 3, 2020-03-11T22:40:55.490 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_osds": 3, 2020-03-11T22:40:55.490 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:40:55.490 INFO:tasks.workunit.client.0.smithi145.stdout: "kb": 314572800, 2020-03-11T22:40:55.491 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used": 3179328, 2020-03-11T22:40:55.491 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_data": 33280, 2020-03-11T22:40:55.491 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_omap": 13, 2020-03-11T22:40:55.491 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_meta": 3145714, 2020-03-11T22:40:55.491 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_avail": 311393472, 2020-03-11T22:40:55.491 INFO:tasks.workunit.client.0.smithi145.stdout: "statfs": { 2020-03-11T22:40:55.491 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 322122547200, 2020-03-11T22:40:55.492 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 318866915328, 2020-03-11T22:40:55.492 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 3221225472, 2020-03-11T22:40:55.492 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 34078720, 2020-03-11T22:40:55.492 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 407206, 2020-03-11T22:40:55.492 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2020-03-11T22:40:55.492 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2020-03-11T22:40:55.493 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2020-03-11T22:40:55.493 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 13752, 2020-03-11T22:40:55.493 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 3221211720 2020-03-11T22:40:55.493 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.493 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_peers": [], 2020-03-11T22:40:55.493 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:40:55.493 INFO:tasks.workunit.client.0.smithi145.stdout: "num_snap_trimming": 0, 2020-03-11T22:40:55.494 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shards_repaired": 0, 2020-03-11T22:40:55.494 INFO:tasks.workunit.client.0.smithi145.stdout: "op_queue_age_hist": { 2020-03-11T22:40:55.494 INFO:tasks.workunit.client.0.smithi145.stdout: "histogram": [], 2020-03-11T22:40:55.494 INFO:tasks.workunit.client.0.smithi145.stdout: "upper_bound": 1 2020-03-11T22:40:55.494 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.494 INFO:tasks.workunit.client.0.smithi145.stdout: "perf_stat": { 2020-03-11T22:40:55.495 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ms": 0, 2020-03-11T22:40:55.495 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ms": 0, 2020-03-11T22:40:55.495 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ns": 0, 2020-03-11T22:40:55.495 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ns": 0 2020-03-11T22:40:55.495 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.495 INFO:tasks.workunit.client.0.smithi145.stdout: "alerts": [], 2020-03-11T22:40:55.495 INFO:tasks.workunit.client.0.smithi145.stdout: "network_ping_times": [] 2020-03-11T22:40:55.496 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.496 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_sum_by_class": { 2020-03-11T22:40:55.496 INFO:tasks.workunit.client.0.smithi145.stdout: "hdd": { 2020-03-11T22:40:55.496 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 0, 2020-03-11T22:40:55.496 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 0, 2020-03-11T22:40:55.496 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pgs": 3, 2020-03-11T22:40:55.496 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osds": 3, 2020-03-11T22:40:55.497 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_osds": 3, 2020-03-11T22:40:55.497 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:40:55.497 INFO:tasks.workunit.client.0.smithi145.stdout: "kb": 314572800, 2020-03-11T22:40:55.497 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used": 3179328, 2020-03-11T22:40:55.497 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_data": 33280, 2020-03-11T22:40:55.498 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_omap": 13, 2020-03-11T22:40:55.498 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_meta": 3145714, 2020-03-11T22:40:55.498 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_avail": 311393472, 2020-03-11T22:40:55.498 INFO:tasks.workunit.client.0.smithi145.stdout: "statfs": { 2020-03-11T22:40:55.498 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 322122547200, 2020-03-11T22:40:55.498 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 318866915328, 2020-03-11T22:40:55.498 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 3221225472, 2020-03-11T22:40:55.499 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 34078720, 2020-03-11T22:40:55.499 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 407206, 2020-03-11T22:40:55.499 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2020-03-11T22:40:55.499 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2020-03-11T22:40:55.499 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2020-03-11T22:40:55.500 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 13752, 2020-03-11T22:40:55.500 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 3221211720 2020-03-11T22:40:55.500 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.500 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_peers": [], 2020-03-11T22:40:55.500 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:40:55.500 INFO:tasks.workunit.client.0.smithi145.stdout: "num_snap_trimming": 0, 2020-03-11T22:40:55.501 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shards_repaired": 0, 2020-03-11T22:40:55.501 INFO:tasks.workunit.client.0.smithi145.stdout: "op_queue_age_hist": { 2020-03-11T22:40:55.501 INFO:tasks.workunit.client.0.smithi145.stdout: "histogram": [], 2020-03-11T22:40:55.501 INFO:tasks.workunit.client.0.smithi145.stdout: "upper_bound": 1 2020-03-11T22:40:55.501 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.501 INFO:tasks.workunit.client.0.smithi145.stdout: "perf_stat": { 2020-03-11T22:40:55.501 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ms": 0, 2020-03-11T22:40:55.502 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ms": 0, 2020-03-11T22:40:55.502 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ns": 0, 2020-03-11T22:40:55.502 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ns": 0 2020-03-11T22:40:55.502 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.502 INFO:tasks.workunit.client.0.smithi145.stdout: "alerts": [], 2020-03-11T22:40:55.502 INFO:tasks.workunit.client.0.smithi145.stdout: "network_ping_times": [] 2020-03-11T22:40:55.502 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:40:55.503 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.503 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_stats": [ 2020-03-11T22:40:55.503 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.503 INFO:tasks.workunit.client.0.smithi145.stdout: "poolid": 1, 2020-03-11T22:40:55.503 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg": 1, 2020-03-11T22:40:55.504 INFO:tasks.workunit.client.0.smithi145.stdout: "stat_sum": { 2020-03-11T22:40:55.504 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes": 48300, 2020-03-11T22:40:55.504 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects": 100, 2020-03-11T22:40:55.504 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_clones": 0, 2020-03-11T22:40:55.504 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_copies": 300, 2020-03-11T22:40:55.504 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:40:55.504 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing": 0, 2020-03-11T22:40:55.504 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_degraded": 100, 2020-03-11T22:40:55.505 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_misplaced": 0, 2020-03-11T22:40:55.505 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_unfound": 0, 2020-03-11T22:40:55.505 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_dirty": 100, 2020-03-11T22:40:55.505 INFO:tasks.workunit.client.0.smithi145.stdout: "num_whiteouts": 0, 2020-03-11T22:40:55.505 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read": 0, 2020-03-11T22:40:55.505 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read_kb": 0, 2020-03-11T22:40:55.506 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write": 100, 2020-03-11T22:40:55.506 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write_kb": 100, 2020-03-11T22:40:55.506 INFO:tasks.workunit.client.0.smithi145.stdout: "num_scrub_errors": 0, 2020-03-11T22:40:55.506 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:40:55.506 INFO:tasks.workunit.client.0.smithi145.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:40:55.506 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_recovered": 0, 2020-03-11T22:40:55.507 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_recovered": 0, 2020-03-11T22:40:55.507 INFO:tasks.workunit.client.0.smithi145.stdout: "num_keys_recovered": 0, 2020-03-11T22:40:55.507 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_omap": 0, 2020-03-11T22:40:55.507 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:40:55.507 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:40:55.507 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush": 0, 2020-03-11T22:40:55.508 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_kb": 0, 2020-03-11T22:40:55.508 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict": 0, 2020-03-11T22:40:55.508 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_kb": 0, 2020-03-11T22:40:55.508 INFO:tasks.workunit.client.0.smithi145.stdout: "num_promote": 0, 2020-03-11T22:40:55.508 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_high": 0, 2020-03-11T22:40:55.508 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_low": 0, 2020-03-11T22:40:55.508 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_some": 0, 2020-03-11T22:40:55.509 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_full": 0, 2020-03-11T22:40:55.509 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_pinned": 0, 2020-03-11T22:40:55.509 INFO:tasks.workunit.client.0.smithi145.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:40:55.509 INFO:tasks.workunit.client.0.smithi145.stdout: "num_large_omap_objects": 0, 2020-03-11T22:40:55.509 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_manifest": 0, 2020-03-11T22:40:55.509 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_bytes": 0, 2020-03-11T22:40:55.510 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_keys": 0, 2020-03-11T22:40:55.510 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_repaired": 0 2020-03-11T22:40:55.510 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.510 INFO:tasks.workunit.client.0.smithi145.stdout: "store_stats": { 2020-03-11T22:40:55.510 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 0, 2020-03-11T22:40:55.510 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 0, 2020-03-11T22:40:55.510 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2020-03-11T22:40:55.511 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 19529728, 2020-03-11T22:40:55.511 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 143228, 2020-03-11T22:40:55.511 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2020-03-11T22:40:55.511 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2020-03-11T22:40:55.511 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2020-03-11T22:40:55.511 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 0, 2020-03-11T22:40:55.512 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 0 2020-03-11T22:40:55.512 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.512 INFO:tasks.workunit.client.0.smithi145.stdout: "log_size": 16, 2020-03-11T22:40:55.512 INFO:tasks.workunit.client.0.smithi145.stdout: "ondisk_log_size": 16, 2020-03-11T22:40:55.512 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 2, 2020-03-11T22:40:55.512 INFO:tasks.workunit.client.0.smithi145.stdout: "acting": 2, 2020-03-11T22:40:55.512 INFO:tasks.workunit.client.0.smithi145.stdout: "num_store_stats": 3 2020-03-11T22:40:55.513 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:40:55.513 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:40:55.513 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_stats": [ 2020-03-11T22:40:55.513 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.513 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2020-03-11T22:40:55.513 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 146028888074 2020-03-11T22:40:55.513 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.514 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.514 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2020-03-11T22:40:55.514 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 42949672974 2020-03-11T22:40:55.514 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.514 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.514 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2020-03-11T22:40:55.515 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 150323855370 2020-03-11T22:40:55.515 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:40:55.515 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:40:55.515 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_by_state": [ 2020-03-11T22:40:55.515 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.515 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "active+undersized+degraded", 2020-03-11T22:40:55.515 INFO:tasks.workunit.client.0.smithi145.stdout: "num": 1 2020-03-11T22:40:55.516 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:40:55.516 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:40:55.516 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_by_osd": [ 2020-03-11T22:40:55.516 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.516 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2020-03-11T22:40:55.516 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 0, 2020-03-11T22:40:55.516 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 1, 2020-03-11T22:40:55.517 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:40:55.517 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.517 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.517 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2020-03-11T22:40:55.517 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 0, 2020-03-11T22:40:55.517 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 0, 2020-03-11T22:40:55.518 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:40:55.518 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.518 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.518 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2020-03-11T22:40:55.518 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 1, 2020-03-11T22:40:55.518 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 1, 2020-03-11T22:40:55.518 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:40:55.519 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:40:55.519 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:40:55.519 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snaps": [ 2020-03-11T22:40:55.519 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:40:55.519 INFO:tasks.workunit.client.0.smithi145.stdout: "pool": 1, 2020-03-11T22:40:55.519 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snaps": {} 2020-03-11T22:40:55.519 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:40:55.520 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:40:55.520 INFO:tasks.workunit.client.0.smithi145.stdout: "servicemap": { 2020-03-11T22:40:55.520 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 1, 2020-03-11T22:40:55.520 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2020-03-11T22:39:18.080018+0000", 2020-03-11T22:40:55.520 INFO:tasks.workunit.client.0.smithi145.stdout: "services": {} 2020-03-11T22:40:55.520 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:40:55.520 INFO:tasks.workunit.client.0.smithi145.stdout: "paxos": { 2020-03-11T22:40:55.521 INFO:tasks.workunit.client.0.smithi145.stdout: "first_committed": 1, 2020-03-11T22:40:55.521 INFO:tasks.workunit.client.0.smithi145.stdout: "last_committed": 302, 2020-03-11T22:40:55.521 INFO:tasks.workunit.client.0.smithi145.stdout: "last_pn": 0, 2020-03-11T22:40:55.521 INFO:tasks.workunit.client.0.smithi145.stdout: "accepted_pn": 0 2020-03-11T22:40:55.521 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:40:55.521 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:40:55.523 INFO:tasks.workunit.client.0.smithi145.stderr:report 657391423 2020-03-11T22:40:55.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:40:55.523 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: expr 2 + 2 2020-03-11T22:40:55.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: objname=existing_4 2020-03-11T22:40:55.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:529: TEST_divergent_2: echo 'writing non-divergent object existing_4' 2020-03-11T22:40:55.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:530: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:40:55.524 INFO:tasks.workunit.client.0.smithi145.stdout:writing non-divergent object existing_4 2020-03-11T22:40:55.674 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:40:55.675 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2020-03-11T22:40:34.921649+0000 28'100 40:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:39.216542+0000 0'0 2020-03-11T22:39:39.216542+0000 0 2020-03-11T22:40:55.675 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:40:55.675 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:40:55.676 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:40:55.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:531: TEST_divergent_2: rados -p test put existing_4 /etc/resolv.conf 2020-03-11T22:40:55.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:40:55.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: wait_for_clean 2020-03-11T22:40:55.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:40:55.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:40:55.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:40:55.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:40:55.731 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:40:55.731 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:40:55.731 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:40:55.731 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:40:55.731 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:40:55.731 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:40:55.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:40:55.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:40:55.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:40:55.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:40:55.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:40:56.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:40:56.132 INFO:tasks.workunit.client.0.smithi145.stderr:1 2020-03-11T22:40:56.133 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2020-03-11T22:40:56.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:40:56.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:40:56.133 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:40:56.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=146028888077 2020-03-11T22:40:56.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888077' 2020-03-11T22:40:56.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:40:56.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:40:56.377 INFO:tasks.workunit.client.0.smithi145.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:40:56.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:40:56.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888077 1-' 2020-03-11T22:40:56.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:40:56.386 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:40:56.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=150323855372 2020-03-11T22:40:56.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888077 1- 2-150323855372' 2020-03-11T22:40:56.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:40:56.515 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-146028888077 2020-03-11T22:40:56.515 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:40:56.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:40:56.516 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-146028888077 2020-03-11T22:40:56.516 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:40:56.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=146028888077 2020-03-11T22:40:56.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 146028888077' 2020-03-11T22:40:56.517 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:40:56.517 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 146028888077 2020-03-11T22:40:56.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 146028888075 -lt 146028888077 2020-03-11T22:40:56.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:40:57.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:40:57.860 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:40:58.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 146028888077 -lt 146028888077 2020-03-11T22:40:58.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:40:58.218 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:40:58.219 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:40:58.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:40:58.219 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:40:58.220 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:40:58.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:40:58.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:40:58.221 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:40:58.221 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 2020-03-11T22:40:58.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672974 -lt 2020-03-11T22:40:58.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672974: unary operator expected 2020-03-11T22:40:58.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:40:58.566 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-150323855372 2020-03-11T22:40:58.567 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:40:58.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:40:58.569 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-150323855372 2020-03-11T22:40:58.569 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:40:58.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=150323855372 2020-03-11T22:40:58.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 150323855372' 2020-03-11T22:40:58.570 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:40:58.571 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 150323855372 2020-03-11T22:40:58.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855373 -lt 150323855372 2020-03-11T22:40:58.924 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:40:58.924 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:58.924 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:59.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:40:59.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:59.369 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:59.369 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:59.370 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:59.370 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:59.370 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:59.370 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:59.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:40:59.701 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:59.701 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:59.701 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:00.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:41:00.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:41:00.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:41:00.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:41:00.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:00.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:41:00.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:00.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:00.246 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:00.247 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:00.247 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:00.247 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:00.247 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:00.247 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:00.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:00.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:00.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:00.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:01.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:41:01.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:01.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:01.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:01.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:01.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:01.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:01.018 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:01.019 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:01.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:01.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:01.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:41:01.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:01.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:41:01.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:01.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:01.658 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:01.659 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:01.659 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:01.659 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:01.659 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:01.659 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:01.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:01.991 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:01.991 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:01.992 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:02.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:41:02.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:02.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:02.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:02.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:02.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:02.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:02.437 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:02.437 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:02.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:02.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:02.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:41:02.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:02.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:41:03.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:03.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:03.284 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:03.284 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:03.284 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:03.284 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:03.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:03.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:03.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:03.612 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:03.612 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:03.613 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:04.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:41:04.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:04.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:04.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:04.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:04.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:04.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:04.055 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:04.055 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:04.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:04.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:04.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:41:04.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:04.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:41:05.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:05.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:05.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:05.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:05.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:05.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:05.318 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:05.318 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:05.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:05.654 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:05.654 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:05.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:06.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:41:06.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:06.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:06.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:06.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:06.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:06.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:06.111 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:06.111 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:06.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:06.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:06.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:41:06.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:06.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:41:08.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:08.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:08.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:08.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:08.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:08.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:08.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:08.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:08.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:08.493 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:08.493 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:08.493 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:08.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:41:08.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:08.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:08.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:08.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:08.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:08.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:08.939 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:08.939 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:09.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:09.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:09.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:41:09.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:09.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:41:12.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:12.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:12.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:12.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:12.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:12.588 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:12.588 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:12.588 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:12.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:12.927 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:12.928 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:12.928 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:13.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:41:13.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:13.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:13.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:13.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:13.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:13.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:13.373 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:13.373 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:13.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:13.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:13.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:41:13.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:13.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:41:20.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:20.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:20.215 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:20.215 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:20.215 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:20.216 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:20.216 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:20.216 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:20.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:20.556 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:20.556 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:20.556 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:20.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:41:20.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:20.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:20.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:21.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:21.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:21.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:21.000 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:21.000 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:21.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:21.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:21.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:41:21.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:21.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:41:28.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:28.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:28.747 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:28.747 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:28.748 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:28.748 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:28.748 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:28.748 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:29.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:29.088 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:29.088 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:29.088 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:29.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:41:29.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:29.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:29.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:29.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:29.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:29.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:29.543 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:29.543 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:29.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:29.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:29.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:41:29.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:41:30.436 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:41:30.436 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_fingerprint": "19898337-fb00-40eb-b748-56bfa6c9fb99", 2020-03-11T22:41:30.436 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:41:30.436 INFO:tasks.workunit.client.0.smithi145.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:41:30.436 INFO:tasks.workunit.client.0.smithi145.stdout: "timestamp": "2020-03-11T22:41:30.432709+0000", 2020-03-11T22:41:30.437 INFO:tasks.workunit.client.0.smithi145.stdout: "tag": "", 2020-03-11T22:41:30.437 INFO:tasks.workunit.client.0.smithi145.stdout: "health": { 2020-03-11T22:41:30.437 INFO:tasks.workunit.client.0.smithi145.stdout: "status": "HEALTH_WARN", 2020-03-11T22:41:30.437 INFO:tasks.workunit.client.0.smithi145.stdout: "checks": { 2020-03-11T22:41:30.437 INFO:tasks.workunit.client.0.smithi145.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:41:30.438 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:41:30.438 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2020-03-11T22:41:30.438 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:41:30.438 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 17 2020-03-11T22:41:30.438 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.438 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [], 2020-03-11T22:41:30.439 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2020-03-11T22:41:30.439 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.439 INFO:tasks.workunit.client.0.smithi145.stdout: "OSD_DOWN": { 2020-03-11T22:41:30.439 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:41:30.439 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2020-03-11T22:41:30.439 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "1 osds down", 2020-03-11T22:41:30.439 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 1 2020-03-11T22:41:30.440 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.440 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2020-03-11T22:41:30.440 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.440 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "osd.1 (root=default,host=smithi145) is down" 2020-03-11T22:41:30.440 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:41:30.440 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:41:30.441 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2020-03-11T22:41:30.441 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.441 INFO:tasks.workunit.client.0.smithi145.stdout: "PG_DEGRADED": { 2020-03-11T22:41:30.441 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:41:30.441 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2020-03-11T22:41:30.442 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded, 1 pg undersized", 2020-03-11T22:41:30.442 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 2 2020-03-11T22:41:30.442 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.442 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2020-03-11T22:41:30.442 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.442 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.0 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:41:30.443 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:41:30.443 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:41:30.443 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2020-03-11T22:41:30.443 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.443 INFO:tasks.workunit.client.0.smithi145.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:41:30.443 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:41:30.444 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2020-03-11T22:41:30.444 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:41:30.444 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 1 2020-03-11T22:41:30.444 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.444 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2020-03-11T22:41:30.444 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.445 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:41:30.445 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.445 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.445 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:41:30.445 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:41:30.445 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:41:30.446 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2020-03-11T22:41:30.446 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:41:30.446 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.446 INFO:tasks.workunit.client.0.smithi145.stdout: "mutes": [] 2020-03-11T22:41:30.446 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.446 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap_first_committed": 1, 2020-03-11T22:41:30.447 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap_last_committed": 1, 2020-03-11T22:41:30.447 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap": { 2020-03-11T22:41:30.447 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 1, 2020-03-11T22:41:30.447 INFO:tasks.workunit.client.0.smithi145.stdout: "fsid": "e665ac71-d8d3-412f-b706-8f1d72482f80", 2020-03-11T22:41:30.447 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2020-03-11T22:39:17.502728Z", 2020-03-11T22:41:30.448 INFO:tasks.workunit.client.0.smithi145.stdout: "created": "2020-03-11T22:39:17.502728Z", 2020-03-11T22:41:30.448 INFO:tasks.workunit.client.0.smithi145.stdout: "min_mon_release": 15, 2020-03-11T22:41:30.448 INFO:tasks.workunit.client.0.smithi145.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:41:30.448 INFO:tasks.workunit.client.0.smithi145.stdout: "features": { 2020-03-11T22:41:30.449 INFO:tasks.workunit.client.0.smithi145.stdout: "persistent": [ 2020-03-11T22:41:30.449 INFO:tasks.workunit.client.0.smithi145.stdout: "kraken", 2020-03-11T22:41:30.449 INFO:tasks.workunit.client.0.smithi145.stdout: "luminous", 2020-03-11T22:41:30.449 INFO:tasks.workunit.client.0.smithi145.stdout: "mimic", 2020-03-11T22:41:30.449 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap-prune", 2020-03-11T22:41:30.449 INFO:tasks.workunit.client.0.smithi145.stdout: "nautilus", 2020-03-11T22:41:30.449 INFO:tasks.workunit.client.0.smithi145.stdout: "octopus" 2020-03-11T22:41:30.450 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:41:30.450 INFO:tasks.workunit.client.0.smithi145.stdout: "optional": [] 2020-03-11T22:41:30.450 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.450 INFO:tasks.workunit.client.0.smithi145.stdout: "mons": [ 2020-03-11T22:41:30.450 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.450 INFO:tasks.workunit.client.0.smithi145.stdout: "rank": 0, 2020-03-11T22:41:30.451 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "a", 2020-03-11T22:41:30.451 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2020-03-11T22:41:30.451 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:41:30.451 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.451 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:41:30.451 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:41:30.452 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 0 2020-03-11T22:41:30.452 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:41:30.452 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:41:30.452 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.452 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:41:30.452 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:41:30.453 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 0, 2020-03-11T22:41:30.453 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 0 2020-03-11T22:41:30.453 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:41:30.453 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:41:30.463 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.463 INFO:tasks.workunit.client.0.smithi145.stdout: "quorum": [ 2020-03-11T22:41:30.464 INFO:tasks.workunit.client.0.smithi145.stdout: 0 2020-03-11T22:41:30.464 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:41:30.464 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap": { 2020-03-11T22:41:30.464 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 40, 2020-03-11T22:41:30.465 INFO:tasks.workunit.client.0.smithi145.stdout: "fsid": "e665ac71-d8d3-412f-b706-8f1d72482f80", 2020-03-11T22:41:30.465 INFO:tasks.workunit.client.0.smithi145.stdout: "created": "2020-03-11T22:39:18.079691+0000", 2020-03-11T22:41:30.465 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2020-03-11T22:40:19.317859+0000", 2020-03-11T22:41:30.465 INFO:tasks.workunit.client.0.smithi145.stdout: "last_up_change": "2020-03-11T22:40:17.555998+0000", 2020-03-11T22:41:30.465 INFO:tasks.workunit.client.0.smithi145.stdout: "last_in_change": "2020-03-11T22:40:19.021132+0000", 2020-03-11T22:41:30.465 INFO:tasks.workunit.client.0.smithi145.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:41:30.465 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_num": 5800384, 2020-03-11T22:41:30.466 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_set": [ 2020-03-11T22:41:30.466 INFO:tasks.workunit.client.0.smithi145.stdout: "nodown", 2020-03-11T22:41:30.466 INFO:tasks.workunit.client.0.smithi145.stdout: "noin", 2020-03-11T22:41:30.466 INFO:tasks.workunit.client.0.smithi145.stdout: "noout", 2020-03-11T22:41:30.466 INFO:tasks.workunit.client.0.smithi145.stdout: "pglog_hardlimit", 2020-03-11T22:41:30.466 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snapdirs", 2020-03-11T22:41:30.466 INFO:tasks.workunit.client.0.smithi145.stdout: "recovery_deletes", 2020-03-11T22:41:30.466 INFO:tasks.workunit.client.0.smithi145.stdout: "sortbitwise" 2020-03-11T22:41:30.467 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:41:30.467 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_version": 7, 2020-03-11T22:41:30.467 INFO:tasks.workunit.client.0.smithi145.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:41:30.467 INFO:tasks.workunit.client.0.smithi145.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:41:30.467 INFO:tasks.workunit.client.0.smithi145.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:41:30.467 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_snapshot": "", 2020-03-11T22:41:30.467 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_max": 1, 2020-03-11T22:41:30.468 INFO:tasks.workunit.client.0.smithi145.stdout: "max_osd": 3, 2020-03-11T22:41:30.468 INFO:tasks.workunit.client.0.smithi145.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:41:30.468 INFO:tasks.workunit.client.0.smithi145.stdout: "min_compat_client": "jewel", 2020-03-11T22:41:30.468 INFO:tasks.workunit.client.0.smithi145.stdout: "require_osd_release": "octopus", 2020-03-11T22:41:30.468 INFO:tasks.workunit.client.0.smithi145.stdout: "pools": [ 2020-03-11T22:41:30.468 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.468 INFO:tasks.workunit.client.0.smithi145.stdout: "pool": 1, 2020-03-11T22:41:30.469 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_name": "test", 2020-03-11T22:41:30.469 INFO:tasks.workunit.client.0.smithi145.stdout: "create_time": "2020-03-11T22:39:39.166414+0000", 2020-03-11T22:41:30.469 INFO:tasks.workunit.client.0.smithi145.stdout: "flags": 1, 2020-03-11T22:41:30.469 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_names": "hashpspool", 2020-03-11T22:41:30.469 INFO:tasks.workunit.client.0.smithi145.stdout: "type": 1, 2020-03-11T22:41:30.469 INFO:tasks.workunit.client.0.smithi145.stdout: "size": 3, 2020-03-11T22:41:30.469 INFO:tasks.workunit.client.0.smithi145.stdout: "min_size": 2, 2020-03-11T22:41:30.469 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_rule": 0, 2020-03-11T22:41:30.470 INFO:tasks.workunit.client.0.smithi145.stdout: "object_hash": 2, 2020-03-11T22:41:30.470 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_autoscale_mode": "off", 2020-03-11T22:41:30.470 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num": 1, 2020-03-11T22:41:30.470 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_placement_num": 1, 2020-03-11T22:41:30.470 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_placement_num_target": 1, 2020-03-11T22:41:30.470 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num_target": 1, 2020-03-11T22:41:30.470 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num_pending": 1, 2020-03-11T22:41:30.471 INFO:tasks.workunit.client.0.smithi145.stdout: "last_pg_merge_meta": { 2020-03-11T22:41:30.471 INFO:tasks.workunit.client.0.smithi145.stdout: "source_pgid": "0.0", 2020-03-11T22:41:30.471 INFO:tasks.workunit.client.0.smithi145.stdout: "ready_epoch": 0, 2020-03-11T22:41:30.471 INFO:tasks.workunit.client.0.smithi145.stdout: "last_epoch_started": 0, 2020-03-11T22:41:30.471 INFO:tasks.workunit.client.0.smithi145.stdout: "last_epoch_clean": 0, 2020-03-11T22:41:30.471 INFO:tasks.workunit.client.0.smithi145.stdout: "source_version": "0'0", 2020-03-11T22:41:30.471 INFO:tasks.workunit.client.0.smithi145.stdout: "target_version": "0'0" 2020-03-11T22:41:30.471 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.472 INFO:tasks.workunit.client.0.smithi145.stdout: "last_change": "28", 2020-03-11T22:41:30.472 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend": "0", 2020-03-11T22:41:30.472 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:41:30.472 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend_preluminous": "0", 2020-03-11T22:41:30.472 INFO:tasks.workunit.client.0.smithi145.stdout: "auid": 0, 2020-03-11T22:41:30.472 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_mode": "selfmanaged", 2020-03-11T22:41:30.472 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_seq": 0, 2020-03-11T22:41:30.472 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_epoch": 0, 2020-03-11T22:41:30.473 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_snaps": [], 2020-03-11T22:41:30.473 INFO:tasks.workunit.client.0.smithi145.stdout: "removed_snaps": "[]", 2020-03-11T22:41:30.473 INFO:tasks.workunit.client.0.smithi145.stdout: "quota_max_bytes": 0, 2020-03-11T22:41:30.473 INFO:tasks.workunit.client.0.smithi145.stdout: "quota_max_objects": 0, 2020-03-11T22:41:30.473 INFO:tasks.workunit.client.0.smithi145.stdout: "tiers": [], 2020-03-11T22:41:30.473 INFO:tasks.workunit.client.0.smithi145.stdout: "tier_of": -1, 2020-03-11T22:41:30.473 INFO:tasks.workunit.client.0.smithi145.stdout: "read_tier": -1, 2020-03-11T22:41:30.474 INFO:tasks.workunit.client.0.smithi145.stdout: "write_tier": -1, 2020-03-11T22:41:30.474 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_mode": "none", 2020-03-11T22:41:30.474 INFO:tasks.workunit.client.0.smithi145.stdout: "target_max_bytes": 0, 2020-03-11T22:41:30.474 INFO:tasks.workunit.client.0.smithi145.stdout: "target_max_objects": 0, 2020-03-11T22:41:30.474 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:41:30.474 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:41:30.475 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:41:30.475 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_min_flush_age": 0, 2020-03-11T22:41:30.475 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_min_evict_age": 0, 2020-03-11T22:41:30.475 INFO:tasks.workunit.client.0.smithi145.stdout: "erasure_code_profile": "", 2020-03-11T22:41:30.475 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_params": { 2020-03-11T22:41:30.475 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "none" 2020-03-11T22:41:30.476 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.476 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_period": 0, 2020-03-11T22:41:30.476 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_count": 0, 2020-03-11T22:41:30.476 INFO:tasks.workunit.client.0.smithi145.stdout: "use_gmt_hitset": true, 2020-03-11T22:41:30.476 INFO:tasks.workunit.client.0.smithi145.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:41:30.476 INFO:tasks.workunit.client.0.smithi145.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:41:30.476 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:41:30.477 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:41:30.477 INFO:tasks.workunit.client.0.smithi145.stdout: "grade_table": [], 2020-03-11T22:41:30.477 INFO:tasks.workunit.client.0.smithi145.stdout: "stripe_width": 0, 2020-03-11T22:41:30.477 INFO:tasks.workunit.client.0.smithi145.stdout: "expected_num_objects": 0, 2020-03-11T22:41:30.477 INFO:tasks.workunit.client.0.smithi145.stdout: "fast_read": false, 2020-03-11T22:41:30.477 INFO:tasks.workunit.client.0.smithi145.stdout: "options": {}, 2020-03-11T22:41:30.477 INFO:tasks.workunit.client.0.smithi145.stdout: "application_metadata": {} 2020-03-11T22:41:30.477 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:41:30.478 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:41:30.478 INFO:tasks.workunit.client.0.smithi145.stdout: "osds": [ 2020-03-11T22:41:30.478 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.478 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2020-03-11T22:41:30.478 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "6548d5c2-3fe9-45a6-8404-614e80715021", 2020-03-11T22:41:30.478 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 1, 2020-03-11T22:41:30.478 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2020-03-11T22:41:30.478 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2020-03-11T22:41:30.479 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2020-03-11T22:41:30.479 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 5, 2020-03-11T22:41:30.479 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 28, 2020-03-11T22:41:30.479 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 34, 2020-03-11T22:41:30.479 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 39, 2020-03-11T22:41:30.479 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 29, 2020-03-11T22:41:30.479 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2020-03-11T22:41:30.480 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2020-03-11T22:41:30.480 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:41:30.480 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.480 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:41:30.480 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:41:30.480 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 30195 2020-03-11T22:41:30.480 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.481 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.481 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:41:30.481 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:41:30.481 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 30195 2020-03-11T22:41:30.481 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:41:30.481 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:41:30.481 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.482 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2020-03-11T22:41:30.482 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:41:30.482 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.482 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:41:30.482 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:41:30.482 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 30195 2020-03-11T22:41:30.482 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.483 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.483 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:41:30.483 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:41:30.483 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 30195 2020-03-11T22:41:30.483 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:41:30.483 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:41:30.483 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.483 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2020-03-11T22:41:30.484 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:41:30.484 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.484 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:41:30.484 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:41:30.484 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 30195 2020-03-11T22:41:30.485 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.485 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.485 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:41:30.485 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:41:30.485 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 30195 2020-03-11T22:41:30.485 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:41:30.485 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:41:30.485 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.486 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2020-03-11T22:41:30.486 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:41:30.486 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.486 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:41:30.486 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:41:30.486 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 30195 2020-03-11T22:41:30.486 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.487 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.487 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:41:30.487 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:41:30.487 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 30195 2020-03-11T22:41:30.487 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:41:30.487 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:41:30.487 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.487 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6803/30195", 2020-03-11T22:41:30.488 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6805/30195", 2020-03-11T22:41:30.488 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6809/30195", 2020-03-11T22:41:30.488 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6807/30195", 2020-03-11T22:41:30.488 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2020-03-11T22:41:30.488 INFO:tasks.workunit.client.0.smithi145.stdout: "exists", 2020-03-11T22:41:30.488 INFO:tasks.workunit.client.0.smithi145.stdout: "up" 2020-03-11T22:41:30.488 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:41:30.488 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.489 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.489 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2020-03-11T22:41:30.489 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "fc159c74-e2d8-44ac-b32a-c54afc816878", 2020-03-11T22:41:30.489 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 0, 2020-03-11T22:41:30.489 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2020-03-11T22:41:30.489 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2020-03-11T22:41:30.489 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2020-03-11T22:41:30.490 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 0, 2020-03-11T22:41:30.490 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 0, 2020-03-11T22:41:30.490 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 10, 2020-03-11T22:41:30.490 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 22, 2020-03-11T22:41:30.490 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 30, 2020-03-11T22:41:30.490 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2020-03-11T22:41:30.490 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2020-03-11T22:41:30.490 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:41:30.491 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.491 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:41:30.491 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:41:30.491 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 25828 2020-03-11T22:41:30.491 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.491 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.491 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:41:30.492 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:41:30.492 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 25828 2020-03-11T22:41:30.492 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:41:30.492 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:41:30.492 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.492 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2020-03-11T22:41:30.492 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:41:30.492 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.493 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:41:30.493 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:41:30.493 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 25828 2020-03-11T22:41:30.493 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.493 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.493 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:41:30.493 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:41:30.494 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 25828 2020-03-11T22:41:30.494 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:41:30.494 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:41:30.494 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.494 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2020-03-11T22:41:30.494 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:41:30.495 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.495 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:41:30.495 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:41:30.495 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 25828 2020-03-11T22:41:30.495 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.495 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.495 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:41:30.496 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:41:30.496 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 25828 2020-03-11T22:41:30.496 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:41:30.496 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:41:30.496 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.496 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2020-03-11T22:41:30.496 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:41:30.496 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.497 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:41:30.497 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:41:30.497 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 25828 2020-03-11T22:41:30.497 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.497 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.497 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:41:30.497 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:41:30.498 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 25828 2020-03-11T22:41:30.498 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:41:30.498 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:41:30.498 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.498 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6811/25828", 2020-03-11T22:41:30.498 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6813/25828", 2020-03-11T22:41:30.498 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6817/25828", 2020-03-11T22:41:30.498 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6815/25828", 2020-03-11T22:41:30.499 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2020-03-11T22:41:30.499 INFO:tasks.workunit.client.0.smithi145.stdout: "exists" 2020-03-11T22:41:30.499 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:41:30.499 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.499 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.499 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2020-03-11T22:41:30.499 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "cc077be5-2bbd-488f-9867-ebfa384fe3f6", 2020-03-11T22:41:30.500 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 1, 2020-03-11T22:41:30.500 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2020-03-11T22:41:30.500 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2020-03-11T22:41:30.500 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2020-03-11T22:41:30.500 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 15, 2020-03-11T22:41:30.500 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 28, 2020-03-11T22:41:30.500 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 35, 2020-03-11T22:41:30.501 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 0, 2020-03-11T22:41:30.501 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 31, 2020-03-11T22:41:30.501 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2020-03-11T22:41:30.501 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2020-03-11T22:41:30.501 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:41:30.501 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.501 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:41:30.501 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:41:30.502 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 30389 2020-03-11T22:41:30.502 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.502 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.502 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:41:30.502 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:41:30.502 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 30389 2020-03-11T22:41:30.503 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:41:30.503 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:41:30.503 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.503 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2020-03-11T22:41:30.503 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:41:30.503 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.503 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:41:30.503 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:41:30.504 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 30389 2020-03-11T22:41:30.504 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.504 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.504 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:41:30.504 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:41:30.504 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 30389 2020-03-11T22:41:30.504 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:41:30.505 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:41:30.505 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.505 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2020-03-11T22:41:30.505 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:41:30.505 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.506 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:41:30.506 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:41:30.506 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 30389 2020-03-11T22:41:30.506 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.506 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.506 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:41:30.506 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:41:30.506 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 30389 2020-03-11T22:41:30.507 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:41:30.507 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:41:30.507 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.507 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2020-03-11T22:41:30.507 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:41:30.507 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.508 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:41:30.508 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:41:30.508 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 30389 2020-03-11T22:41:30.508 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.508 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.508 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:41:30.508 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:41:30.508 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 30389 2020-03-11T22:41:30.509 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:41:30.509 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:41:30.509 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.509 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6811/30389", 2020-03-11T22:41:30.509 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6813/30389", 2020-03-11T22:41:30.509 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6817/30389", 2020-03-11T22:41:30.509 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6815/30389", 2020-03-11T22:41:30.510 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2020-03-11T22:41:30.510 INFO:tasks.workunit.client.0.smithi145.stdout: "exists", 2020-03-11T22:41:30.510 INFO:tasks.workunit.client.0.smithi145.stdout: "up" 2020-03-11T22:41:30.510 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:41:30.510 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:41:30.510 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:41:30.510 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_xinfo": [ 2020-03-11T22:41:30.510 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.511 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2020-03-11T22:41:30.511 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "2020-03-11T22:40:11.564170+0000", 2020-03-11T22:41:30.511 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2020-03-11T22:41:30.511 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2020-03-11T22:41:30.511 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138292836696063, 2020-03-11T22:41:30.511 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2020-03-11T22:41:30.511 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2020-03-11T22:39:24.632044+0000", 2020-03-11T22:41:30.511 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2020-03-11T22:41:30.512 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.512 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.512 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2020-03-11T22:41:30.512 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "2020-03-11T22:40:12.041763+0000", 2020-03-11T22:41:30.512 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2020-03-11T22:41:30.512 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2020-03-11T22:41:30.512 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138292836696063, 2020-03-11T22:41:30.513 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2020-03-11T22:41:30.513 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2020-03-11T22:39:30.875922+0000", 2020-03-11T22:41:30.513 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2020-03-11T22:41:30.513 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.513 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.513 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2020-03-11T22:41:30.514 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "2020-03-11T22:40:12.508032+0000", 2020-03-11T22:41:30.514 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2020-03-11T22:41:30.514 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2020-03-11T22:41:30.514 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138292836696063, 2020-03-11T22:41:30.514 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2020-03-11T22:41:30.514 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2020-03-11T22:39:37.114069+0000", 2020-03-11T22:41:30.514 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2020-03-11T22:41:30.514 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:41:30.515 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:41:30.515 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_upmap": [], 2020-03-11T22:41:30.515 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_upmap_items": [], 2020-03-11T22:41:30.515 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_temp": [], 2020-03-11T22:41:30.515 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_temp": [], 2020-03-11T22:41:30.515 INFO:tasks.workunit.client.0.smithi145.stdout: "blacklist": {}, 2020-03-11T22:41:30.516 INFO:tasks.workunit.client.0.smithi145.stdout: "erasure_code_profiles": { 2020-03-11T22:41:30.516 INFO:tasks.workunit.client.0.smithi145.stdout: "default": { 2020-03-11T22:41:30.516 INFO:tasks.workunit.client.0.smithi145.stdout: "k": "2", 2020-03-11T22:41:30.516 INFO:tasks.workunit.client.0.smithi145.stdout: "m": "2", 2020-03-11T22:41:30.516 INFO:tasks.workunit.client.0.smithi145.stdout: "plugin": "jerasure", 2020-03-11T22:41:30.516 INFO:tasks.workunit.client.0.smithi145.stdout: "technique": "reed_sol_van" 2020-03-11T22:41:30.516 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:41:30.517 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.517 INFO:tasks.workunit.client.0.smithi145.stdout: "removed_snaps_queue": [], 2020-03-11T22:41:30.517 INFO:tasks.workunit.client.0.smithi145.stdout: "new_removed_snaps": [], 2020-03-11T22:41:30.517 INFO:tasks.workunit.client.0.smithi145.stdout: "new_purged_snaps": [], 2020-03-11T22:41:30.517 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_node_flags": {}, 2020-03-11T22:41:30.517 INFO:tasks.workunit.client.0.smithi145.stdout: "device_class_flags": {} 2020-03-11T22:41:30.517 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.517 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_metadata": [ 2020-03-11T22:41:30.518 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.518 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2020-03-11T22:41:30.518 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2020-03-11T22:41:30.518 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6804/30195,v1:127.0.0.1:6805/30195]", 2020-03-11T22:41:30.518 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2020-03-11T22:41:30.518 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2020-03-11T22:41:30.518 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:41:30.519 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:41:30.519 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:41:30.519 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:41:30.519 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:41:30.519 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:41:30.519 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:41:30.519 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:41:30.519 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:41:30.520 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:41:30.520 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:41:30.520 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:41:30.520 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "octopus", 2020-03-11T22:41:30.520 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:41:30.520 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:41:30.520 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:41:30.521 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2020-03-11T22:41:30.521 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XLDP", 2020-03-11T22:41:30.521 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:41:30.521 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2020-03-11T22:41:30.521 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2020-03-11T22:41:30.521 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:41:30.521 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2020-03-11T22:41:30.521 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6802/30195,v1:127.0.0.1:6803/30195]", 2020-03-11T22:41:30.522 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2020-03-11T22:41:30.522 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/30195,v1:127.0.0.1:6809/30195]", 2020-03-11T22:41:30.522 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/30195,v1:127.0.0.1:6807/30195]", 2020-03-11T22:41:30.522 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2020-03-11T22:41:30.522 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2020-03-11T22:41:30.522 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:41:30.522 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:41:30.523 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2020-03-11T22:41:30.523 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32655820", 2020-03-11T22:41:30.523 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:41:30.523 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:41:30.523 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2020-03-11T22:41:30.523 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:41:30.523 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:41:30.523 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2020-03-11T22:41:30.524 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.524 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.524 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2020-03-11T22:41:30.524 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2020-03-11T22:41:30.524 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6812/25828,v1:127.0.0.1:6813/25828]", 2020-03-11T22:41:30.524 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2020-03-11T22:41:30.524 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2020-03-11T22:41:30.524 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:41:30.525 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:41:30.525 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:41:30.525 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:41:30.525 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:41:30.525 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:41:30.526 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:41:30.526 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:41:30.526 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:41:30.526 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:41:30.526 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:41:30.526 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:41:30.527 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "octopus", 2020-03-11T22:41:30.527 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:41:30.527 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:41:30.527 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:41:30.527 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2020-03-11T22:41:30.528 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XLDP", 2020-03-11T22:41:30.528 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:41:30.528 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2020-03-11T22:41:30.528 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2020-03-11T22:41:30.528 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:41:30.528 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2020-03-11T22:41:30.528 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6810/25828,v1:127.0.0.1:6811/25828]", 2020-03-11T22:41:30.529 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2020-03-11T22:41:30.529 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/25828,v1:127.0.0.1:6817/25828]", 2020-03-11T22:41:30.529 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/25828,v1:127.0.0.1:6815/25828]", 2020-03-11T22:41:30.529 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2020-03-11T22:41:30.529 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2020-03-11T22:41:30.529 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:41:30.529 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:41:30.530 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2020-03-11T22:41:30.530 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32655820", 2020-03-11T22:41:30.530 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:41:30.530 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:41:30.530 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2020-03-11T22:41:30.530 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:41:30.530 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:41:30.530 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2020-03-11T22:41:30.531 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.531 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.531 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2020-03-11T22:41:30.531 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2020-03-11T22:41:30.531 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6812/30389,v1:127.0.0.1:6813/30389]", 2020-03-11T22:41:30.531 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2020-03-11T22:41:30.531 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2020-03-11T22:41:30.532 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:41:30.532 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:41:30.532 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:41:30.532 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:41:30.532 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:41:30.532 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:41:30.532 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:41:30.532 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:41:30.533 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:41:30.533 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:41:30.533 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:41:30.533 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:41:30.533 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "octopus", 2020-03-11T22:41:30.533 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:41:30.533 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:41:30.534 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:41:30.534 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2020-03-11T22:41:30.534 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XLDP", 2020-03-11T22:41:30.534 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:41:30.534 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2020-03-11T22:41:30.534 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2020-03-11T22:41:30.534 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:41:30.534 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2020-03-11T22:41:30.535 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6810/30389,v1:127.0.0.1:6811/30389]", 2020-03-11T22:41:30.535 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2020-03-11T22:41:30.535 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/30389,v1:127.0.0.1:6817/30389]", 2020-03-11T22:41:30.535 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/30389,v1:127.0.0.1:6815/30389]", 2020-03-11T22:41:30.535 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2020-03-11T22:41:30.535 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2020-03-11T22:41:30.536 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:41:30.536 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:41:30.536 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2020-03-11T22:41:30.536 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32655820", 2020-03-11T22:41:30.536 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:41:30.536 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:41:30.536 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2020-03-11T22:41:30.537 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:41:30.537 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:41:30.537 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2020-03-11T22:41:30.537 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:41:30.537 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:41:30.537 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap_first_committed": 1, 2020-03-11T22:41:30.537 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap_last_committed": 40, 2020-03-11T22:41:30.537 INFO:tasks.workunit.client.0.smithi145.stdout: "crushmap": { 2020-03-11T22:41:30.538 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": [ 2020-03-11T22:41:30.538 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.538 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2020-03-11T22:41:30.538 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.0", 2020-03-11T22:41:30.538 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2020-03-11T22:41:30.538 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.538 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.538 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2020-03-11T22:41:30.539 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.1", 2020-03-11T22:41:30.539 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2020-03-11T22:41:30.539 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.539 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.539 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2020-03-11T22:41:30.539 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.2", 2020-03-11T22:41:30.539 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2020-03-11T22:41:30.539 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:41:30.540 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:41:30.540 INFO:tasks.workunit.client.0.smithi145.stdout: "types": [ 2020-03-11T22:41:30.540 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.540 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 0, 2020-03-11T22:41:30.540 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd" 2020-03-11T22:41:30.540 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.540 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.540 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2020-03-11T22:41:30.541 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "host" 2020-03-11T22:41:30.541 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.541 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.541 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 2, 2020-03-11T22:41:30.541 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "chassis" 2020-03-11T22:41:30.541 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.541 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.542 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 3, 2020-03-11T22:41:30.542 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "rack" 2020-03-11T22:41:30.542 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.542 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.542 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 4, 2020-03-11T22:41:30.542 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "row" 2020-03-11T22:41:30.543 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.543 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.543 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 5, 2020-03-11T22:41:30.543 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "pdu" 2020-03-11T22:41:30.543 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.543 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.543 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 6, 2020-03-11T22:41:30.544 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "pod" 2020-03-11T22:41:30.544 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.544 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.544 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 7, 2020-03-11T22:41:30.544 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "room" 2020-03-11T22:41:30.544 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.544 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.544 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 8, 2020-03-11T22:41:30.545 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "datacenter" 2020-03-11T22:41:30.545 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.545 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.545 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 9, 2020-03-11T22:41:30.545 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "zone" 2020-03-11T22:41:30.545 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.545 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.545 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 10, 2020-03-11T22:41:30.546 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "region" 2020-03-11T22:41:30.546 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.546 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.546 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2020-03-11T22:41:30.546 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "root" 2020-03-11T22:41:30.547 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:41:30.547 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:41:30.547 INFO:tasks.workunit.client.0.smithi145.stdout: "buckets": [ 2020-03-11T22:41:30.547 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.547 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -1, 2020-03-11T22:41:30.547 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "default", 2020-03-11T22:41:30.547 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2020-03-11T22:41:30.547 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "root", 2020-03-11T22:41:30.548 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2020-03-11T22:41:30.548 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2020-03-11T22:41:30.548 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2020-03-11T22:41:30.548 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:41:30.548 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.548 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -3, 2020-03-11T22:41:30.548 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2020-03-11T22:41:30.548 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2020-03-11T22:41:30.549 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:41:30.549 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:41:30.549 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.549 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.549 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -2, 2020-03-11T22:41:30.549 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "default~hdd", 2020-03-11T22:41:30.549 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2020-03-11T22:41:30.550 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "root", 2020-03-11T22:41:30.550 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2020-03-11T22:41:30.550 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2020-03-11T22:41:30.550 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2020-03-11T22:41:30.550 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:41:30.550 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.550 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -4, 2020-03-11T22:41:30.550 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2020-03-11T22:41:30.551 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2020-03-11T22:41:30.551 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:41:30.551 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:41:30.551 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.551 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.551 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -3, 2020-03-11T22:41:30.551 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "smithi145", 2020-03-11T22:41:30.551 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2020-03-11T22:41:30.552 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "host", 2020-03-11T22:41:30.552 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2020-03-11T22:41:30.552 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2020-03-11T22:41:30.552 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2020-03-11T22:41:30.552 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:41:30.552 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.552 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2020-03-11T22:41:30.553 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2020-03-11T22:41:30.553 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2020-03-11T22:41:30.553 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.553 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.553 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2020-03-11T22:41:30.554 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2020-03-11T22:41:30.554 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 1 2020-03-11T22:41:30.554 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.554 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.554 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2020-03-11T22:41:30.554 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2020-03-11T22:41:30.555 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 2 2020-03-11T22:41:30.555 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:41:30.555 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:41:30.555 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.555 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.556 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -4, 2020-03-11T22:41:30.557 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "smithi145~hdd", 2020-03-11T22:41:30.557 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2020-03-11T22:41:30.557 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "host", 2020-03-11T22:41:30.557 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2020-03-11T22:41:30.558 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2020-03-11T22:41:30.558 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2020-03-11T22:41:30.558 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:41:30.558 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.558 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2020-03-11T22:41:30.559 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2020-03-11T22:41:30.559 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2020-03-11T22:41:30.559 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.559 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.559 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2020-03-11T22:41:30.560 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2020-03-11T22:41:30.560 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 1 2020-03-11T22:41:30.560 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.560 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.560 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2020-03-11T22:41:30.561 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2020-03-11T22:41:30.561 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 2 2020-03-11T22:41:30.561 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:41:30.561 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:41:30.562 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:41:30.562 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:41:30.562 INFO:tasks.workunit.client.0.smithi145.stdout: "rules": [ 2020-03-11T22:41:30.562 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.562 INFO:tasks.workunit.client.0.smithi145.stdout: "rule_id": 0, 2020-03-11T22:41:30.562 INFO:tasks.workunit.client.0.smithi145.stdout: "rule_name": "replicated_rule", 2020-03-11T22:41:30.563 INFO:tasks.workunit.client.0.smithi145.stdout: "ruleset": 0, 2020-03-11T22:41:30.563 INFO:tasks.workunit.client.0.smithi145.stdout: "type": 1, 2020-03-11T22:41:30.563 INFO:tasks.workunit.client.0.smithi145.stdout: "min_size": 1, 2020-03-11T22:41:30.564 INFO:tasks.workunit.client.0.smithi145.stdout: "max_size": 10, 2020-03-11T22:41:30.564 INFO:tasks.workunit.client.0.smithi145.stdout: "steps": [ 2020-03-11T22:41:30.564 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.564 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "take", 2020-03-11T22:41:30.565 INFO:tasks.workunit.client.0.smithi145.stdout: "item": -1, 2020-03-11T22:41:30.565 INFO:tasks.workunit.client.0.smithi145.stdout: "item_name": "default" 2020-03-11T22:41:30.565 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.565 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.565 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "choose_firstn", 2020-03-11T22:41:30.566 INFO:tasks.workunit.client.0.smithi145.stdout: "num": 0, 2020-03-11T22:41:30.566 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "osd" 2020-03-11T22:41:30.566 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.566 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.568 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "emit" 2020-03-11T22:41:30.568 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:41:30.568 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:41:30.569 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:41:30.569 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:41:30.569 INFO:tasks.workunit.client.0.smithi145.stdout: "tunables": { 2020-03-11T22:41:30.569 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_local_tries": 0, 2020-03-11T22:41:30.570 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:41:30.570 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_total_tries": 50, 2020-03-11T22:41:30.570 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:41:30.570 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:41:30.570 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_stable": 1, 2020-03-11T22:41:30.571 INFO:tasks.workunit.client.0.smithi145.stdout: "straw_calc_version": 1, 2020-03-11T22:41:30.571 INFO:tasks.workunit.client.0.smithi145.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:41:30.571 INFO:tasks.workunit.client.0.smithi145.stdout: "profile": "jewel", 2020-03-11T22:41:30.571 INFO:tasks.workunit.client.0.smithi145.stdout: "optimal_tunables": 1, 2020-03-11T22:41:30.571 INFO:tasks.workunit.client.0.smithi145.stdout: "legacy_tunables": 0, 2020-03-11T22:41:30.571 INFO:tasks.workunit.client.0.smithi145.stdout: "minimum_required_version": "jewel", 2020-03-11T22:41:30.571 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables": 1, 2020-03-11T22:41:30.571 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables2": 1, 2020-03-11T22:41:30.572 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v2_rules": 0, 2020-03-11T22:41:30.572 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables3": 1, 2020-03-11T22:41:30.572 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v3_rules": 0, 2020-03-11T22:41:30.572 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v4_buckets": 1, 2020-03-11T22:41:30.572 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables5": 1, 2020-03-11T22:41:30.572 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v5_rules": 0 2020-03-11T22:41:30.572 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.572 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_args": {} 2020-03-11T22:41:30.573 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.573 INFO:tasks.workunit.client.0.smithi145.stdout: "fsmap": { 2020-03-11T22:41:30.573 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 1, 2020-03-11T22:41:30.573 INFO:tasks.workunit.client.0.smithi145.stdout: "default_fscid": -1, 2020-03-11T22:41:30.573 INFO:tasks.workunit.client.0.smithi145.stdout: "compat": { 2020-03-11T22:41:30.573 INFO:tasks.workunit.client.0.smithi145.stdout: "compat": {}, 2020-03-11T22:41:30.573 INFO:tasks.workunit.client.0.smithi145.stdout: "ro_compat": {}, 2020-03-11T22:41:30.574 INFO:tasks.workunit.client.0.smithi145.stdout: "incompat": { 2020-03-11T22:41:30.574 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_1": "base v0.20", 2020-03-11T22:41:30.574 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:41:30.574 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:41:30.574 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:41:30.574 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:41:30.574 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:41:30.574 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_8": "no anchor table", 2020-03-11T22:41:30.575 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_9": "file layout v2", 2020-03-11T22:41:30.575 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:41:30.575 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:41:30.575 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.575 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_flags": { 2020-03-11T22:41:30.575 INFO:tasks.workunit.client.0.smithi145.stdout: "enable_multiple": false, 2020-03-11T22:41:30.575 INFO:tasks.workunit.client.0.smithi145.stdout: "ever_enabled_multiple": false 2020-03-11T22:41:30.576 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.576 INFO:tasks.workunit.client.0.smithi145.stdout: "standbys": [], 2020-03-11T22:41:30.576 INFO:tasks.workunit.client.0.smithi145.stdout: "filesystems": [] 2020-03-11T22:41:30.576 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.576 INFO:tasks.workunit.client.0.smithi145.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:41:30.576 INFO:tasks.workunit.client.0.smithi145.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:41:30.576 INFO:tasks.workunit.client.0.smithi145.stdout: "auth": { 2020-03-11T22:41:30.577 INFO:tasks.workunit.client.0.smithi145.stdout: "first_committed": 1, 2020-03-11T22:41:30.577 INFO:tasks.workunit.client.0.smithi145.stdout: "last_committed": 5, 2020-03-11T22:41:30.577 INFO:tasks.workunit.client.0.smithi145.stdout: "num_secrets": 3 2020-03-11T22:41:30.577 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.577 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg": 1, 2020-03-11T22:41:30.578 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_active": 1, 2020-03-11T22:41:30.578 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_unknown": 0, 2020-03-11T22:41:30.578 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osd": 3, 2020-03-11T22:41:30.578 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_sum": { 2020-03-11T22:41:30.578 INFO:tasks.workunit.client.0.smithi145.stdout: "stat_sum": { 2020-03-11T22:41:30.578 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes": 47947, 2020-03-11T22:41:30.578 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects": 100, 2020-03-11T22:41:30.578 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_clones": 0, 2020-03-11T22:41:30.579 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_copies": 300, 2020-03-11T22:41:30.579 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:41:30.579 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing": 0, 2020-03-11T22:41:30.579 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_degraded": 100, 2020-03-11T22:41:30.579 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_misplaced": 0, 2020-03-11T22:41:30.579 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_unfound": 0, 2020-03-11T22:41:30.579 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_dirty": 100, 2020-03-11T22:41:30.579 INFO:tasks.workunit.client.0.smithi145.stdout: "num_whiteouts": 0, 2020-03-11T22:41:30.580 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read": 0, 2020-03-11T22:41:30.580 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read_kb": 0, 2020-03-11T22:41:30.580 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write": 101, 2020-03-11T22:41:30.580 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write_kb": 101, 2020-03-11T22:41:30.580 INFO:tasks.workunit.client.0.smithi145.stdout: "num_scrub_errors": 0, 2020-03-11T22:41:30.580 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:41:30.580 INFO:tasks.workunit.client.0.smithi145.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:41:30.580 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_recovered": 0, 2020-03-11T22:41:30.581 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_recovered": 0, 2020-03-11T22:41:30.581 INFO:tasks.workunit.client.0.smithi145.stdout: "num_keys_recovered": 0, 2020-03-11T22:41:30.581 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_omap": 0, 2020-03-11T22:41:30.581 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:41:30.581 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:41:30.581 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush": 0, 2020-03-11T22:41:30.581 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_kb": 0, 2020-03-11T22:41:30.582 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict": 0, 2020-03-11T22:41:30.582 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_kb": 0, 2020-03-11T22:41:30.582 INFO:tasks.workunit.client.0.smithi145.stdout: "num_promote": 0, 2020-03-11T22:41:30.582 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_high": 0, 2020-03-11T22:41:30.582 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_low": 0, 2020-03-11T22:41:30.582 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_some": 0, 2020-03-11T22:41:30.583 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_full": 0, 2020-03-11T22:41:30.583 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_pinned": 0, 2020-03-11T22:41:30.583 INFO:tasks.workunit.client.0.smithi145.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:41:30.583 INFO:tasks.workunit.client.0.smithi145.stdout: "num_large_omap_objects": 0, 2020-03-11T22:41:30.583 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_manifest": 0, 2020-03-11T22:41:30.583 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_bytes": 0, 2020-03-11T22:41:30.583 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_keys": 0, 2020-03-11T22:41:30.583 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_repaired": 0 2020-03-11T22:41:30.584 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.584 INFO:tasks.workunit.client.0.smithi145.stdout: "store_stats": { 2020-03-11T22:41:30.584 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 0, 2020-03-11T22:41:30.584 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 0, 2020-03-11T22:41:30.584 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2020-03-11T22:41:30.584 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 0, 2020-03-11T22:41:30.584 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 0, 2020-03-11T22:41:30.585 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2020-03-11T22:41:30.585 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2020-03-11T22:41:30.585 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2020-03-11T22:41:30.585 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 0, 2020-03-11T22:41:30.585 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 0 2020-03-11T22:41:30.585 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.585 INFO:tasks.workunit.client.0.smithi145.stdout: "log_size": 17, 2020-03-11T22:41:30.585 INFO:tasks.workunit.client.0.smithi145.stdout: "ondisk_log_size": 17, 2020-03-11T22:41:30.586 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 2, 2020-03-11T22:41:30.586 INFO:tasks.workunit.client.0.smithi145.stdout: "acting": 2, 2020-03-11T22:41:30.586 INFO:tasks.workunit.client.0.smithi145.stdout: "num_store_stats": 0 2020-03-11T22:41:30.586 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.586 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_sum": { 2020-03-11T22:41:30.586 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 0, 2020-03-11T22:41:30.586 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 0, 2020-03-11T22:41:30.586 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pgs": 3, 2020-03-11T22:41:30.587 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osds": 3, 2020-03-11T22:41:30.587 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_osds": 3, 2020-03-11T22:41:30.587 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:41:30.587 INFO:tasks.workunit.client.0.smithi145.stdout: "kb": 314572800, 2020-03-11T22:41:30.587 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used": 3179328, 2020-03-11T22:41:30.588 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_data": 33280, 2020-03-11T22:41:30.588 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_omap": 13, 2020-03-11T22:41:30.588 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_meta": 3145714, 2020-03-11T22:41:30.588 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_avail": 311393472, 2020-03-11T22:41:30.588 INFO:tasks.workunit.client.0.smithi145.stdout: "statfs": { 2020-03-11T22:41:30.588 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 322122547200, 2020-03-11T22:41:30.588 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 318866915328, 2020-03-11T22:41:30.588 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 3221225472, 2020-03-11T22:41:30.589 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 34078720, 2020-03-11T22:41:30.589 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 406500, 2020-03-11T22:41:30.589 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2020-03-11T22:41:30.589 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2020-03-11T22:41:30.589 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2020-03-11T22:41:30.589 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 13752, 2020-03-11T22:41:30.589 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 3221211720 2020-03-11T22:41:30.590 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.590 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_peers": [], 2020-03-11T22:41:30.590 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:41:30.590 INFO:tasks.workunit.client.0.smithi145.stdout: "num_snap_trimming": 0, 2020-03-11T22:41:30.590 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shards_repaired": 0, 2020-03-11T22:41:30.590 INFO:tasks.workunit.client.0.smithi145.stdout: "op_queue_age_hist": { 2020-03-11T22:41:30.590 INFO:tasks.workunit.client.0.smithi145.stdout: "histogram": [], 2020-03-11T22:41:30.590 INFO:tasks.workunit.client.0.smithi145.stdout: "upper_bound": 1 2020-03-11T22:41:30.591 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.591 INFO:tasks.workunit.client.0.smithi145.stdout: "perf_stat": { 2020-03-11T22:41:30.591 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ms": 0, 2020-03-11T22:41:30.591 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ms": 0, 2020-03-11T22:41:30.591 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ns": 0, 2020-03-11T22:41:30.591 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ns": 0 2020-03-11T22:41:30.591 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.592 INFO:tasks.workunit.client.0.smithi145.stdout: "alerts": [], 2020-03-11T22:41:30.592 INFO:tasks.workunit.client.0.smithi145.stdout: "network_ping_times": [] 2020-03-11T22:41:30.592 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.592 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_sum_by_class": { 2020-03-11T22:41:30.592 INFO:tasks.workunit.client.0.smithi145.stdout: "hdd": { 2020-03-11T22:41:30.592 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 0, 2020-03-11T22:41:30.592 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 0, 2020-03-11T22:41:30.592 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pgs": 3, 2020-03-11T22:41:30.593 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osds": 3, 2020-03-11T22:41:30.593 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_osds": 3, 2020-03-11T22:41:30.593 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:41:30.593 INFO:tasks.workunit.client.0.smithi145.stdout: "kb": 314572800, 2020-03-11T22:41:30.593 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used": 3179328, 2020-03-11T22:41:30.593 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_data": 33280, 2020-03-11T22:41:30.593 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_omap": 13, 2020-03-11T22:41:30.593 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_meta": 3145714, 2020-03-11T22:41:30.594 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_avail": 311393472, 2020-03-11T22:41:30.594 INFO:tasks.workunit.client.0.smithi145.stdout: "statfs": { 2020-03-11T22:41:30.594 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 322122547200, 2020-03-11T22:41:30.594 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 318866915328, 2020-03-11T22:41:30.594 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 3221225472, 2020-03-11T22:41:30.594 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 34078720, 2020-03-11T22:41:30.594 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 406500, 2020-03-11T22:41:30.595 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2020-03-11T22:41:30.595 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2020-03-11T22:41:30.595 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2020-03-11T22:41:30.595 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 13752, 2020-03-11T22:41:30.595 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 3221211720 2020-03-11T22:41:30.595 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.595 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_peers": [], 2020-03-11T22:41:30.595 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:41:30.596 INFO:tasks.workunit.client.0.smithi145.stdout: "num_snap_trimming": 0, 2020-03-11T22:41:30.596 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shards_repaired": 0, 2020-03-11T22:41:30.596 INFO:tasks.workunit.client.0.smithi145.stdout: "op_queue_age_hist": { 2020-03-11T22:41:30.596 INFO:tasks.workunit.client.0.smithi145.stdout: "histogram": [], 2020-03-11T22:41:30.596 INFO:tasks.workunit.client.0.smithi145.stdout: "upper_bound": 1 2020-03-11T22:41:30.596 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.596 INFO:tasks.workunit.client.0.smithi145.stdout: "perf_stat": { 2020-03-11T22:41:30.596 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ms": 0, 2020-03-11T22:41:30.597 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ms": 0, 2020-03-11T22:41:30.597 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ns": 0, 2020-03-11T22:41:30.597 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ns": 0 2020-03-11T22:41:30.597 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.597 INFO:tasks.workunit.client.0.smithi145.stdout: "alerts": [], 2020-03-11T22:41:30.598 INFO:tasks.workunit.client.0.smithi145.stdout: "network_ping_times": [] 2020-03-11T22:41:30.598 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:41:30.598 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.598 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_stats": [ 2020-03-11T22:41:30.598 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.598 INFO:tasks.workunit.client.0.smithi145.stdout: "poolid": 1, 2020-03-11T22:41:30.598 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg": 1, 2020-03-11T22:41:30.598 INFO:tasks.workunit.client.0.smithi145.stdout: "stat_sum": { 2020-03-11T22:41:30.599 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes": 47947, 2020-03-11T22:41:30.599 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects": 100, 2020-03-11T22:41:30.599 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_clones": 0, 2020-03-11T22:41:30.599 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_copies": 300, 2020-03-11T22:41:30.599 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:41:30.599 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing": 0, 2020-03-11T22:41:30.599 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_degraded": 100, 2020-03-11T22:41:30.599 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_misplaced": 0, 2020-03-11T22:41:30.600 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_unfound": 0, 2020-03-11T22:41:30.600 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_dirty": 100, 2020-03-11T22:41:30.600 INFO:tasks.workunit.client.0.smithi145.stdout: "num_whiteouts": 0, 2020-03-11T22:41:30.600 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read": 0, 2020-03-11T22:41:30.600 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read_kb": 0, 2020-03-11T22:41:30.600 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write": 101, 2020-03-11T22:41:30.600 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write_kb": 101, 2020-03-11T22:41:30.601 INFO:tasks.workunit.client.0.smithi145.stdout: "num_scrub_errors": 0, 2020-03-11T22:41:30.601 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:41:30.601 INFO:tasks.workunit.client.0.smithi145.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:41:30.601 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_recovered": 0, 2020-03-11T22:41:30.601 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_recovered": 0, 2020-03-11T22:41:30.601 INFO:tasks.workunit.client.0.smithi145.stdout: "num_keys_recovered": 0, 2020-03-11T22:41:30.601 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_omap": 0, 2020-03-11T22:41:30.602 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:41:30.602 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:41:30.602 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush": 0, 2020-03-11T22:41:30.602 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_kb": 0, 2020-03-11T22:41:30.602 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict": 0, 2020-03-11T22:41:30.602 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_kb": 0, 2020-03-11T22:41:30.602 INFO:tasks.workunit.client.0.smithi145.stdout: "num_promote": 0, 2020-03-11T22:41:30.603 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_high": 0, 2020-03-11T22:41:30.603 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_low": 0, 2020-03-11T22:41:30.603 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_some": 0, 2020-03-11T22:41:30.603 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_full": 0, 2020-03-11T22:41:30.603 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_pinned": 0, 2020-03-11T22:41:30.603 INFO:tasks.workunit.client.0.smithi145.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:41:30.603 INFO:tasks.workunit.client.0.smithi145.stdout: "num_large_omap_objects": 0, 2020-03-11T22:41:30.604 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_manifest": 0, 2020-03-11T22:41:30.604 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_bytes": 0, 2020-03-11T22:41:30.604 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_keys": 0, 2020-03-11T22:41:30.604 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_repaired": 0 2020-03-11T22:41:30.604 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.604 INFO:tasks.workunit.client.0.smithi145.stdout: "store_stats": { 2020-03-11T22:41:30.604 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 0, 2020-03-11T22:41:30.604 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 0, 2020-03-11T22:41:30.605 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2020-03-11T22:41:30.605 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 19529728, 2020-03-11T22:41:30.605 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 142522, 2020-03-11T22:41:30.605 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2020-03-11T22:41:30.605 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2020-03-11T22:41:30.605 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2020-03-11T22:41:30.605 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 0, 2020-03-11T22:41:30.605 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 0 2020-03-11T22:41:30.606 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.606 INFO:tasks.workunit.client.0.smithi145.stdout: "log_size": 17, 2020-03-11T22:41:30.606 INFO:tasks.workunit.client.0.smithi145.stdout: "ondisk_log_size": 17, 2020-03-11T22:41:30.606 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 2, 2020-03-11T22:41:30.606 INFO:tasks.workunit.client.0.smithi145.stdout: "acting": 2, 2020-03-11T22:41:30.606 INFO:tasks.workunit.client.0.smithi145.stdout: "num_store_stats": 3 2020-03-11T22:41:30.606 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:41:30.606 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:41:30.607 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_stats": [ 2020-03-11T22:41:30.607 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.607 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2020-03-11T22:41:30.607 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 146028888083 2020-03-11T22:41:30.607 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.607 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.608 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2020-03-11T22:41:30.608 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 42949672974 2020-03-11T22:41:30.608 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.608 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.608 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2020-03-11T22:41:30.608 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 150323855379 2020-03-11T22:41:30.609 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:41:30.609 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:41:30.609 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_by_state": [ 2020-03-11T22:41:30.609 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.610 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "active+undersized+degraded", 2020-03-11T22:41:30.610 INFO:tasks.workunit.client.0.smithi145.stdout: "num": 1 2020-03-11T22:41:30.610 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:41:30.610 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:41:30.610 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_by_osd": [ 2020-03-11T22:41:30.611 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.611 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2020-03-11T22:41:30.611 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 0, 2020-03-11T22:41:30.611 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 1, 2020-03-11T22:41:30.611 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:41:30.611 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.611 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.611 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2020-03-11T22:41:30.612 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 0, 2020-03-11T22:41:30.612 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 0, 2020-03-11T22:41:30.612 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:41:30.612 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.612 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.612 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2020-03-11T22:41:30.612 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 1, 2020-03-11T22:41:30.612 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 1, 2020-03-11T22:41:30.613 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:41:30.613 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:41:30.613 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:41:30.613 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snaps": [ 2020-03-11T22:41:30.613 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:41:30.613 INFO:tasks.workunit.client.0.smithi145.stdout: "pool": 1, 2020-03-11T22:41:30.613 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snaps": {} 2020-03-11T22:41:30.614 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:41:30.614 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:41:30.614 INFO:tasks.workunit.client.0.smithi145.stdout: "servicemap": { 2020-03-11T22:41:30.614 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 1, 2020-03-11T22:41:30.614 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2020-03-11T22:39:18.080018+0000", 2020-03-11T22:41:30.614 INFO:tasks.workunit.client.0.smithi145.stdout: "services": {} 2020-03-11T22:41:30.615 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:41:30.615 INFO:tasks.workunit.client.0.smithi145.stdout: "paxos": { 2020-03-11T22:41:30.615 INFO:tasks.workunit.client.0.smithi145.stdout: "first_committed": 1, 2020-03-11T22:41:30.615 INFO:tasks.workunit.client.0.smithi145.stdout: "last_committed": 379, 2020-03-11T22:41:30.615 INFO:tasks.workunit.client.0.smithi145.stdout: "last_pn": 0, 2020-03-11T22:41:30.615 INFO:tasks.workunit.client.0.smithi145.stdout: "accepted_pn": 0 2020-03-11T22:41:30.615 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:41:30.615 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:41:30.616 INFO:tasks.workunit.client.0.smithi145.stderr:report 328147483 2020-03-11T22:41:30.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:41:30.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:536: TEST_divergent_2: echo 'delay recovery' 2020-03-11T22:41:30.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:537: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:41:30.617 INFO:tasks.workunit.client.0.smithi145.stdout:delay recovery 2020-03-11T22:41:30.766 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:41:30.766 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:40:34.921649+0000 40'101 40:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:39.216542+0000 0'0 2020-03-11T22:39:39.216542+0000 0 2020-03-11T22:41:30.766 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:41:30.766 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:41:30.767 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:41:30.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:41:30.777 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.0 2020-03-11T22:41:30.777 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:41:30.777 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:41:30.777 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:41:30.777 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:41:30.777 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:41:30.778 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17402/ceph-osd.0.asok 2020-03-11T22:41:30.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:41:30.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17402/ceph-osd.0.asok set_recovery_delay 100000 2020-03-11T22:41:30.868 INFO:tasks.workunit.client.0.smithi145.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:41:30.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:41:30.876 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.2 2020-03-11T22:41:30.876 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:41:30.876 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:41:30.877 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:41:30.877 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:41:30.877 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:41:30.877 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17402/ceph-osd.2.asok 2020-03-11T22:41:30.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:41:30.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17402/ceph-osd.2.asok set_recovery_delay 100000 2020-03-11T22:41:30.967 INFO:tasks.workunit.client.0.smithi145.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:41:30.974 INFO:tasks.workunit.client.0.smithi145.stdout:revive divergent 1 2020-03-11T22:41:30.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:544: TEST_divergent_2: echo 'revive divergent 1' 2020-03-11T22:41:30.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:545: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:41:31.293 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:41:31.293 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:40:34.921649+0000 40'101 40:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:39.216542+0000 0'0 2020-03-11T22:39:39.216542+0000 0 2020-03-11T22:41:31.294 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:41:31.294 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:41:31.295 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:41:31.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:546: TEST_divergent_2: ceph osd set noup 2020-03-11T22:41:32.033 INFO:tasks.workunit.client.0.smithi145.stderr:noup is set 2020-03-11T22:41:32.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:547: TEST_divergent_2: activate_osd td/divergent-priors 1 2020-03-11T22:41:32.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:41:32.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:41:32.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:41:32.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:41:32.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:41:32.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:41:32.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:41:32.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:41:32.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:41:32.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:41:32.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:41:32.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:41:32.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:41:32.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:41:32.046 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:41:32.046 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:41:32.046 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:41:32.047 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:41:32.047 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:41:32.047 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:41:32.047 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:41:32.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:41:32.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:41:32.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:41:32.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:41:32.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:41:32.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:41:32.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:41:32.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:41:32.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:41:32.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:41:32.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:41:32.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:41:32.049 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:41:32.049 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2020-03-11T22:41:32.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:41:32.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:41:32.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:41:32.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:41:32.070 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:41:32.067+0000 7fbdd7018ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:32.072 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:41:32.069+0000 7fbdd7018ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:32.073 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:41:32.070+0000 7fbdd7018ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:32.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: return 0 2020-03-11T22:41:32.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:548: TEST_divergent_2: sleep 5 2020-03-11T22:41:32.588 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:41:32.586+0000 7fbdd7018ec0 -1 Falling back to public interface 2020-03-11T22:41:33.416 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:41:33.414+0000 7fbdd7018ec0 -1 osd.1 28 log_to_monitors {default=true} 2020-03-11T22:41:37.395 INFO:tasks.workunit.client.0.smithi145.stdout:delay recovery divergent 2020-03-11T22:41:37.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:550: TEST_divergent_2: echo 'delay recovery divergent' 2020-03-11T22:41:37.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:551: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:41:37.724 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:41:37.724 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:40:34.921649+0000 40'101 41:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:39.216542+0000 0'0 2020-03-11T22:39:39.216542+0000 0 2020-03-11T22:41:37.724 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:41:37.724 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:41:37.726 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:41:37.735 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: get_asok_path osd.1 2020-03-11T22:41:37.735 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:41:37.735 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:41:37.735 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:41:37.735 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:41:37.735 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:41:37.735 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17402/ceph-osd.1.asok 2020-03-11T22:41:37.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:41:37.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17402/ceph-osd.1.asok set_recovery_delay 100000 2020-03-11T22:41:37.826 INFO:tasks.workunit.client.0.smithi145.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:41:37.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:554: TEST_divergent_2: ceph osd unset noup 2020-03-11T22:41:38.418 INFO:tasks.workunit.client.0.smithi145.stderr:noup is unset 2020-03-11T22:41:38.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:556: TEST_divergent_2: wait_for_osd up 0 2020-03-11T22:41:38.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:41:38.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:41:38.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:41:38.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:41:38.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:41:38.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:41:38.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:41:38.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:41:38.428 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:41:38.778 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 34 up_thru 39 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/30195,v1:127.0.0.1:6803/30195] [v2:127.0.0.1:6804/30195,v1:127.0.0.1:6805/30195] exists,up 6548d5c2-3fe9-45a6-8404-614e80715021 2020-03-11T22:41:38.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:41:38.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:41:38.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:41:38.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:557: TEST_divergent_2: wait_for_osd up 1 2020-03-11T22:41:38.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:41:38.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:41:38.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:41:38.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:41:38.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:41:38.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:41:38.780 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:41:38.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:41:38.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:41:39.117 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 44 up_thru 44 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/32976,v1:127.0.0.1:6819/32976] [v2:127.0.0.1:6820/32976,v1:127.0.0.1:6821/32976] exists,up fc159c74-e2d8-44ac-b32a-c54afc816878 2020-03-11T22:41:39.118 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:41:39.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:41:39.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:41:39.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:41:39.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:558: TEST_divergent_2: wait_for_osd up 2 2020-03-11T22:41:39.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:41:39.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:41:39.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:41:39.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:41:39.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:41:39.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:41:39.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:41:39.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:41:39.467 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 35 up_thru 0 down_at 31 last_clean_interval [15,28) [v2:127.0.0.1:6810/30389,v1:127.0.0.1:6811/30389] [v2:127.0.0.1:6812/30389,v1:127.0.0.1:6813/30389] exists,up cc077be5-2bbd-488f-9867-ebfa384fe3f6 2020-03-11T22:41:39.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:41:39.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:41:39.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:41:39.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:560: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:41:39.789 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:41:39.789 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:40:34.921649+0000 40'101 41:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:39.216542+0000 0'0 2020-03-11T22:39:39.216542+0000 0 2020-03-11T22:41:39.789 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:41:39.790 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:41:39.790 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:41:39.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:561: TEST_divergent_2: echo 'wait for peering' 2020-03-11T22:41:39.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:562: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:41:39.799 INFO:tasks.workunit.client.0.smithi145.stdout:wait for peering 2020-03-11T22:41:40.123 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:41:40.123 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:40:34.921649+0000 40'101 41:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:39:39.216542+0000 0'0 2020-03-11T22:39:39.216542+0000 0 2020-03-11T22:41:40.123 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:41:40.123 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:41:40.124 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:41:40.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:563: TEST_divergent_2: rados -p test put foo /etc/fstab 2020-03-11T22:41:47.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:567: TEST_divergent_2: echo 'killing divergent 1' 2020-03-11T22:41:47.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:568: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:41:47.744 INFO:tasks.workunit.client.0.smithi145.stdout:killing divergent 1 2020-03-11T22:41:48.073 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:41:48.074 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 4 4 0 0 47947 0 0 11 11 active+recovering+degraded+wait 2020-03-11T22:41:38.681709+0000 40'101 45:125 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:39.216542+0000 0'0 2020-03-11T22:39:39.216542+0000 0 2020-03-11T22:41:48.074 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:41:48.074 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:41:48.075 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:41:48.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:569: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd.1 2020-03-11T22:41:48.083 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:41:48.084 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:41:48.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:41:48.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:41:48.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:41:48.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:41:48.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:572: TEST_divergent_2: expfile=td/divergent-priors/exp.17402.out 2020-03-11T22:41:48.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:573: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.17402.out 2020-03-11T22:41:48.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:41:48.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:41:48.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-03-11T22:41:48.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:41:48.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-03-11T22:41:48.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.17402.out 2020-03-11T22:41:48.843 INFO:tasks.workunit.client.0.smithi145.stderr:Exporting 1.0 info 1.0( v 45'102 lc 0'0 (28'90,45'102] local-lis/les=44/45 n=101 ec=22/22 lis/c=44/22 les/c/f=45/23/0 sis=44 pruub=8.989374215s) 2020-03-11T22:41:48.860 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:023dd3d2:::existing_12:head# 2020-03-11T22:41:48.878 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:04c40a0c:::existing_93:head# 2020-03-11T22:41:48.894 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:064ff5fb:::existing_36:head# 2020-03-11T22:41:48.896 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:091f9f1d:::existing_94:head# 2020-03-11T22:41:48.897 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:0a865a93:::existing_54:head# 2020-03-11T22:41:48.903 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:0fed513c:::existing_41:head# 2020-03-11T22:41:48.903 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:13b9e0b8:::existing_92:head# 2020-03-11T22:41:48.904 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:169157fa:::existing_81:head# 2020-03-11T22:41:48.905 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:17731382:::existing_58:head# 2020-03-11T22:41:48.905 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:18a45a39:::existing_43:head# 2020-03-11T22:41:48.906 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:1c160f01:::existing_66:head# 2020-03-11T22:41:48.907 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:1dd6c249:::existing_97:head# 2020-03-11T22:41:48.912 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:227e0c77:::existing_7:head# 2020-03-11T22:41:48.913 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:23e1b063:::existing_59:head# 2020-03-11T22:41:48.914 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:28566edb:::existing_78:head# 2020-03-11T22:41:48.914 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:2a101182:::existing_20:head# 2020-03-11T22:41:48.924 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:2d1519da:::existing_96:head# 2020-03-11T22:41:48.925 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:3078b87a:::existing_86:head# 2020-03-11T22:41:48.925 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:30bde01b:::existing_15:head# 2020-03-11T22:41:48.926 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:317a7e19:::existing_56:head# 2020-03-11T22:41:48.926 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:38b9c7fa:::existing_33:head# 2020-03-11T22:41:48.928 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:3e16d84b:::existing_48:head# 2020-03-11T22:41:48.934 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:45fe8203:::existing_75:head# 2020-03-11T22:41:48.938 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:4e82c9ba:::existing_50:head# 2020-03-11T22:41:48.939 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:5298dcdb:::existing_57:head# 2020-03-11T22:41:48.939 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:52ebaa44:::existing_32:head# 2020-03-11T22:41:48.940 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:53588912:::existing_9:head# 2020-03-11T22:41:48.940 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:5b7e2de6:::existing_84:head# 2020-03-11T22:41:48.941 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:5da0255c:::existing_13:head# 2020-03-11T22:41:48.942 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:5da2dab8:::existing_53:head# 2020-03-11T22:41:48.942 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:5ee3514b:::existing_30:head# 2020-03-11T22:41:48.943 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:602f83fe:::foo:head# 2020-03-11T22:41:48.943 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:60426cc7:::existing_62:head# 2020-03-11T22:41:48.944 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:628918ca:::existing_79:head# 2020-03-11T22:41:48.944 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:63f8ab2d:::existing_77:head# 2020-03-11T22:41:48.945 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:67e1780f:::existing_27:head# 2020-03-11T22:41:48.945 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:68143ca1:::existing_71:head# 2020-03-11T22:41:48.953 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:6bac9509:::existing_29:head# 2020-03-11T22:41:48.954 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:6bbec798:::existing_45:head# 2020-03-11T22:41:48.954 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:6cb23a40:::existing_65:head# 2020-03-11T22:41:48.955 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:7055ec5e:::existing_83:head# 2020-03-11T22:41:48.957 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:70a54e2b:::existing_87:head# 2020-03-11T22:41:48.959 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:750ace21:::existing_19:head# 2020-03-11T22:41:48.965 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:78623b1d:::existing_6:head# 2020-03-11T22:41:48.966 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:79881152:::existing_63:head# 2020-03-11T22:41:48.966 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:7a80c0d5:::existing_99:head# 2020-03-11T22:41:48.967 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:81490677:::existing_76:head# 2020-03-11T22:41:48.968 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:85ec8e5d:::existing_85:head# 2020-03-11T22:41:48.969 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:8e2abbf0:::existing_49:head# 2020-03-11T22:41:48.969 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:8ee0fe0f:::existing_52:head# 2020-03-11T22:41:48.970 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:923c5d4d:::existing_5:head# 2020-03-11T22:41:48.970 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:92731e71:::existing_89:head# 2020-03-11T22:41:48.971 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:928205cf:::existing_82:head# 2020-03-11T22:41:48.972 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:93be2267:::existing_24:head# 2020-03-11T22:41:48.972 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:99ceda2c:::existing_44:head# 2020-03-11T22:41:48.973 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:9dd917ee:::existing_26:head# 2020-03-11T22:41:48.974 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:a28805a6:::existing_72:head# 2020-03-11T22:41:48.974 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:a294a8f6:::existing_16:head# 2020-03-11T22:41:48.975 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:a2f5473a:::existing_42:head# 2020-03-11T22:41:48.975 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:a504a533:::existing_90:head# 2020-03-11T22:41:48.976 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:a691337e:::existing_61:head# 2020-03-11T22:41:48.976 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:a6e8d7ac:::existing_28:head# 2020-03-11T22:41:48.977 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:a9fde86e:::existing_18:head# 2020-03-11T22:41:48.977 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:aa309faa:::existing_68:head# 2020-03-11T22:41:48.978 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:ae40b0ed:::existing_31:head# 2020-03-11T22:41:48.978 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:b279b1fe:::existing_10:head# 2020-03-11T22:41:48.979 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:b343e277:::existing_88:head# 2020-03-11T22:41:48.980 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:b7130e90:::existing_80:head# 2020-03-11T22:41:48.981 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:bc6f7309:::existing_70:head# 2020-03-11T22:41:48.982 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:be8489b7:::existing_38:head# 2020-03-11T22:41:48.982 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:beb4e5e3:::existing_47:head# 2020-03-11T22:41:48.983 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:bebec877:::existing_74:head# 2020-03-11T22:41:48.984 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:c04347d8:::existing_60:head# 2020-03-11T22:41:48.984 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:c156033e:::existing_100:head# 2020-03-11T22:41:48.985 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:c19f5a8e:::existing_8:head# 2020-03-11T22:41:48.985 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:c3e6ebe4:::existing_17:head# 2020-03-11T22:41:48.990 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:c6c5a7c2:::existing_39:head# 2020-03-11T22:41:48.991 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:cc1d0abf:::existing_95:head# 2020-03-11T22:41:48.991 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:ccb3865b:::existing_73:head# 2020-03-11T22:41:48.992 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:cd84e19e:::existing_40:head# 2020-03-11T22:41:48.992 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:d2cd64ce:::existing_37:head# 2020-03-11T22:41:48.993 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:d41a25c9:::existing_22:head# 2020-03-11T22:41:48.993 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:d7f088da:::existing_91:head# 2020-03-11T22:41:48.994 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:d9cbd1d6:::existing_64:head# 2020-03-11T22:41:48.995 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:dcc4ff82:::existing_98:head# 2020-03-11T22:41:48.996 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:dd677bd3:::existing_35:head# 2020-03-11T22:41:48.997 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:dd6d90e7:::existing_46:head# 2020-03-11T22:41:48.997 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:e0e22077:::existing_21:head# 2020-03-11T22:41:48.998 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:e255b710:::existing_55:head# 2020-03-11T22:41:48.998 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:eff1d5db:::existing_14:head# 2020-03-11T22:41:48.999 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:f1b156bf:::existing_23:head# 2020-03-11T22:41:48.999 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:f5ce4199:::existing_34:head# 2020-03-11T22:41:49.000 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:f6809456:::existing_11:head# 2020-03-11T22:41:49.000 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:f699a2a9:::existing_67:head# 2020-03-11T22:41:49.001 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:f9baac43:::existing_25:head# 2020-03-11T22:41:49.002 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:fb76688e:::existing_51:head# 2020-03-11T22:41:49.002 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:fbb92eec:::existing_69:head# 2020-03-11T22:41:49.008 INFO:tasks.workunit.client.0.smithi145.stdout: marking collection for removal 2020-03-11T22:41:49.008 INFO:tasks.workunit.client.0.smithi145.stdout:setting '_remove' omap key 2020-03-11T22:41:49.009 INFO:tasks.workunit.client.0.smithi145.stdout:finish_remove_pgs 1.0_head removing 1.0 2020-03-11T22:41:49.009 INFO:tasks.workunit.client.0.smithi145.stderr:Export successful 2020-03-11T22:41:49.061 INFO:tasks.workunit.client.0.smithi145.stderr:Remove successful 2020-03-11T22:41:49.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:574: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.17402.out 2020-03-11T22:41:49.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:41:49.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:41:49.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-03-11T22:41:49.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:41:49.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-03-11T22:41:49.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.17402.out 2020-03-11T22:41:50.062 INFO:tasks.workunit.client.0.smithi145.stdout:get_pg_num_history pg_num_history pg_num_history(e45 pg_nums {1={22=1}} deleted_pools ) 2020-03-11T22:41:50.062 INFO:tasks.workunit.client.0.smithi145.stdout:Importing pgid 1.0 2020-03-11T22:41:50.062 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:023dd3d2:::existing_12:head# 2020-03-11T22:41:50.063 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.118 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:04c40a0c:::existing_93:head# 2020-03-11T22:41:50.118 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.126 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:064ff5fb:::existing_36:head# 2020-03-11T22:41:50.127 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.135 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:091f9f1d:::existing_94:head# 2020-03-11T22:41:50.135 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.143 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:0a865a93:::existing_54:head# 2020-03-11T22:41:50.143 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.151 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:0fed513c:::existing_41:head# 2020-03-11T22:41:50.152 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.160 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:13b9e0b8:::existing_92:head# 2020-03-11T22:41:50.160 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.168 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:169157fa:::existing_81:head# 2020-03-11T22:41:50.168 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.177 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:17731382:::existing_58:head# 2020-03-11T22:41:50.177 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.185 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:18a45a39:::existing_43:head# 2020-03-11T22:41:50.185 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.193 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:1c160f01:::existing_66:head# 2020-03-11T22:41:50.193 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.202 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:1dd6c249:::existing_97:head# 2020-03-11T22:41:50.202 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.210 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:227e0c77:::existing_7:head# 2020-03-11T22:41:50.210 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.218 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:23e1b063:::existing_59:head# 2020-03-11T22:41:50.218 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.253 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:28566edb:::existing_78:head# 2020-03-11T22:41:50.253 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.257 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:2a101182:::existing_20:head# 2020-03-11T22:41:50.257 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.266 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:2d1519da:::existing_96:head# 2020-03-11T22:41:50.266 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.274 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:3078b87a:::existing_86:head# 2020-03-11T22:41:50.274 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.282 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:30bde01b:::existing_15:head# 2020-03-11T22:41:50.282 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.291 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:317a7e19:::existing_56:head# 2020-03-11T22:41:50.291 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.299 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:38b9c7fa:::existing_33:head# 2020-03-11T22:41:50.299 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.307 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:3e16d84b:::existing_48:head# 2020-03-11T22:41:50.307 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.316 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:45fe8203:::existing_75:head# 2020-03-11T22:41:50.316 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.324 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:4e82c9ba:::existing_50:head# 2020-03-11T22:41:50.324 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.332 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:5298dcdb:::existing_57:head# 2020-03-11T22:41:50.332 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.341 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:52ebaa44:::existing_32:head# 2020-03-11T22:41:50.341 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.349 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:53588912:::existing_9:head# 2020-03-11T22:41:50.349 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.357 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:5b7e2de6:::existing_84:head# 2020-03-11T22:41:50.358 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.394 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:5da0255c:::existing_13:head# 2020-03-11T22:41:50.394 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.402 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:5da2dab8:::existing_53:head# 2020-03-11T22:41:50.402 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.410 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:5ee3514b:::existing_30:head# 2020-03-11T22:41:50.410 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.418 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:602f83fe:::foo:head# 2020-03-11T22:41:50.418 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.427 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:60426cc7:::existing_62:head# 2020-03-11T22:41:50.427 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.435 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:628918ca:::existing_79:head# 2020-03-11T22:41:50.435 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.443 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:63f8ab2d:::existing_77:head# 2020-03-11T22:41:50.444 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.452 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:67e1780f:::existing_27:head# 2020-03-11T22:41:50.452 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.460 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:68143ca1:::existing_71:head# 2020-03-11T22:41:50.460 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.469 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:6bac9509:::existing_29:head# 2020-03-11T22:41:50.469 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.479 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:6bbec798:::existing_45:head# 2020-03-11T22:41:50.479 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.485 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:6cb23a40:::existing_65:head# 2020-03-11T22:41:50.486 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.494 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:7055ec5e:::existing_83:head# 2020-03-11T22:41:50.494 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.528 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:70a54e2b:::existing_87:head# 2020-03-11T22:41:50.528 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.535 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:750ace21:::existing_19:head# 2020-03-11T22:41:50.536 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.544 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:78623b1d:::existing_6:head# 2020-03-11T22:41:50.544 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.552 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:79881152:::existing_63:head# 2020-03-11T22:41:50.552 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.560 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:7a80c0d5:::existing_99:head# 2020-03-11T22:41:50.561 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.569 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:81490677:::existing_76:head# 2020-03-11T22:41:50.569 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.577 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:85ec8e5d:::existing_85:head# 2020-03-11T22:41:50.578 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.585 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:8e2abbf0:::existing_49:head# 2020-03-11T22:41:50.585 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.594 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:8ee0fe0f:::existing_52:head# 2020-03-11T22:41:50.594 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.602 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:923c5d4d:::existing_5:head# 2020-03-11T22:41:50.602 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.611 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:92731e71:::existing_89:head# 2020-03-11T22:41:50.612 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.619 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:928205cf:::existing_82:head# 2020-03-11T22:41:50.619 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.627 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:93be2267:::existing_24:head# 2020-03-11T22:41:50.627 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.636 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:99ceda2c:::existing_44:head# 2020-03-11T22:41:50.636 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.686 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:9dd917ee:::existing_26:head# 2020-03-11T22:41:50.686 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.711 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:a28805a6:::existing_72:head# 2020-03-11T22:41:50.711 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.719 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:a294a8f6:::existing_16:head# 2020-03-11T22:41:50.719 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.727 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:a2f5473a:::existing_42:head# 2020-03-11T22:41:50.727 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.736 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:a504a533:::existing_90:head# 2020-03-11T22:41:50.736 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.744 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:a691337e:::existing_61:head# 2020-03-11T22:41:50.744 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.752 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:a6e8d7ac:::existing_28:head# 2020-03-11T22:41:50.753 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.761 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:a9fde86e:::existing_18:head# 2020-03-11T22:41:50.761 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.769 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:aa309faa:::existing_68:head# 2020-03-11T22:41:50.769 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.778 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:ae40b0ed:::existing_31:head# 2020-03-11T22:41:50.778 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.821 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:b279b1fe:::existing_10:head# 2020-03-11T22:41:50.821 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.853 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:b343e277:::existing_88:head# 2020-03-11T22:41:50.853 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.861 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:b7130e90:::existing_80:head# 2020-03-11T22:41:50.861 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.895 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:bc6f7309:::existing_70:head# 2020-03-11T22:41:50.895 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.903 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:be8489b7:::existing_38:head# 2020-03-11T22:41:50.903 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.911 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:beb4e5e3:::existing_47:head# 2020-03-11T22:41:50.911 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.919 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:bebec877:::existing_74:head# 2020-03-11T22:41:50.919 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.928 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:c04347d8:::existing_60:head# 2020-03-11T22:41:50.928 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.936 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:c156033e:::existing_100:head# 2020-03-11T22:41:50.936 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.944 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:c19f5a8e:::existing_8:head# 2020-03-11T22:41:50.945 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.953 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:c3e6ebe4:::existing_17:head# 2020-03-11T22:41:50.953 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.961 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:c6c5a7c2:::existing_39:head# 2020-03-11T22:41:50.961 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.969 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:cc1d0abf:::existing_95:head# 2020-03-11T22:41:50.970 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.978 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:ccb3865b:::existing_73:head# 2020-03-11T22:41:50.978 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.986 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:cd84e19e:::existing_40:head# 2020-03-11T22:41:50.986 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:50.994 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:d2cd64ce:::existing_37:head# 2020-03-11T22:41:50.995 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:51.003 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:d41a25c9:::existing_22:head# 2020-03-11T22:41:51.003 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:51.036 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:d7f088da:::existing_91:head# 2020-03-11T22:41:51.037 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:51.045 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:d9cbd1d6:::existing_64:head# 2020-03-11T22:41:51.045 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:51.053 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:dcc4ff82:::existing_98:head# 2020-03-11T22:41:51.053 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:51.061 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:dd677bd3:::existing_35:head# 2020-03-11T22:41:51.062 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:51.070 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:dd6d90e7:::existing_46:head# 2020-03-11T22:41:51.070 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:51.078 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:e0e22077:::existing_21:head# 2020-03-11T22:41:51.078 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:51.086 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:e255b710:::existing_55:head# 2020-03-11T22:41:51.086 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:51.095 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:eff1d5db:::existing_14:head# 2020-03-11T22:41:51.095 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:51.103 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:f1b156bf:::existing_23:head# 2020-03-11T22:41:51.103 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:51.111 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:f5ce4199:::existing_34:head# 2020-03-11T22:41:51.111 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:51.120 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:f6809456:::existing_11:head# 2020-03-11T22:41:51.120 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:51.128 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:f699a2a9:::existing_67:head# 2020-03-11T22:41:51.128 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:51.136 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:f9baac43:::existing_25:head# 2020-03-11T22:41:51.137 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:51.145 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:fb76688e:::existing_51:head# 2020-03-11T22:41:51.145 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:51.178 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:fbb92eec:::existing_69:head# 2020-03-11T22:41:51.178 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:41:51.186 INFO:tasks.workunit.client.0.smithi145.stdout:write_pg epoch 45 info 1.0( v 45'102 lc 0'0 (28'90,45'102] local-lis/les=44/45 n=101 ec=22/22 lis/c=44/22 les/c/f=45/23/0 sis=44 pruub=8.989374215s) 2020-03-11T22:41:51.195 INFO:tasks.workunit.client.0.smithi145.stdout:Import successful 2020-03-11T22:41:51.721 INFO:tasks.workunit.client.0.smithi145.stdout:reviving divergent 1 2020-03-11T22:41:51.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:576: TEST_divergent_2: echo 'reviving divergent 1' 2020-03-11T22:41:51.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:577: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:41:52.051 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:41:52.052 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 4 4 0 0 47947 0 0 11 11 active+recovering+degraded+wait 2020-03-11T22:41:38.681709+0000 40'101 45:125 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:39.216542+0000 0'0 2020-03-11T22:39:39.216542+0000 0 2020-03-11T22:41:52.052 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:41:52.052 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:41:52.053 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:41:52.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:578: TEST_divergent_2: activate_osd td/divergent-priors 1 2020-03-11T22:41:52.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:41:52.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:41:52.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:41:52.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:41:52.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:41:52.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:41:52.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:41:52.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:41:52.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:41:52.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:41:52.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:41:52.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:41:52.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:41:52.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:41:52.067 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:41:52.067 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:41:52.067 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:41:52.067 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:41:52.067 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:41:52.067 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:41:52.067 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:41:52.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:41:52.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:41:52.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:41:52.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:41:52.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:41:52.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:41:52.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:41:52.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:41:52.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:41:52.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:41:52.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:41:52.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:41:52.070 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:41:52.070 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2020-03-11T22:41:52.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:41:52.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:41:52.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:41:52.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:41:52.090 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:41:52.087+0000 7fe850ac4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:52.092 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:41:52.088+0000 7fe850ac4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:52.093 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:41:52.090+0000 7fe850ac4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:52.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-03-11T22:41:52.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:41:52.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:41:52.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:41:52.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:41:52.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:41:52.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:41:52.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:41:52.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:41:52.412 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:41:52.640 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:41:52.636+0000 7fe850ac4ec0 -1 Falling back to public interface 2020-03-11T22:41:52.757 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 44 up_thru 44 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/32976,v1:127.0.0.1:6819/32976] [v2:127.0.0.1:6820/32976,v1:127.0.0.1:6821/32976] exists,up fc159c74-e2d8-44ac-b32a-c54afc816878 2020-03-11T22:41:52.757 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:41:52.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:41:52.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:41:52.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:41:52.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:579: TEST_divergent_2: wait_for_osd up 1 2020-03-11T22:41:52.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:41:52.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:41:52.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:41:52.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:41:52.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:41:52.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:41:52.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:41:52.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:41:53.098 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 44 up_thru 44 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/32976,v1:127.0.0.1:6819/32976] [v2:127.0.0.1:6820/32976,v1:127.0.0.1:6821/32976] exists,up fc159c74-e2d8-44ac-b32a-c54afc816878 2020-03-11T22:41:53.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:41:53.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:41:53.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:41:53.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:581: TEST_divergent_2: sleep 20 2020-03-11T22:41:53.404 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:41:53.401+0000 7fe850ac4ec0 -1 osd.1 45 log_to_monitors {default=true} 2020-03-11T22:42:13.099 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: get_asok_path osd.1 2020-03-11T22:42:13.099 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:42:13.100 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:42:13.100 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:42:13.101 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:13.101 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:42:13.101 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17402/ceph-osd.1.asok 2020-03-11T22:42:13.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:42:13.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17402/ceph-osd.1.asok dump_ops_in_flight 2020-03-11T22:42:13.198 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:42:13.199 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [], 2020-03-11T22:42:13.199 INFO:tasks.workunit.client.0.smithi145.stdout: "num_ops": 0 2020-03-11T22:42:13.199 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:42:13.206 INFO:tasks.workunit.client.0.smithi145.stdout:allowing recovery 2020-03-11T22:42:13.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:584: TEST_divergent_2: echo 'allowing recovery' 2020-03-11T22:42:13.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:585: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:42:13.525 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:42:13.525 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 101 0 0 0 0 48430 0 0 12 12 active+clean 2020-03-11T22:41:55.181456+0000 45'102 48:148 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:39.216542+0000 0'0 2020-03-11T22:39:39.216542+0000 0 2020-03-11T22:42:13.525 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:42:13.526 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:42:13.526 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:42:13.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2020-03-11T22:42:13.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.0 debug kick_recovery_wq 0 2020-03-11T22:42:13.655 INFO:tasks.workunit.client.0.smithi145.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:42:13.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2020-03-11T22:42:13.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.1 debug kick_recovery_wq 0 2020-03-11T22:42:13.784 INFO:tasks.workunit.client.0.smithi145.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:42:13.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2020-03-11T22:42:13.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.2 debug kick_recovery_wq 0 2020-03-11T22:42:13.924 INFO:tasks.workunit.client.0.smithi145.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:42:13.932 INFO:tasks.workunit.client.0.smithi145.stdout:reading divergent objects 2020-03-11T22:42:13.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:592: TEST_divergent_2: echo 'reading divergent objects' 2020-03-11T22:42:13.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:593: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:42:14.250 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:42:14.251 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 101 0 0 0 0 48430 0 0 12 12 active+clean 2020-03-11T22:41:55.181456+0000 45'102 48:148 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:39:39.216542+0000 0'0 2020-03-11T22:39:39.216542+0000 0 2020-03-11T22:42:14.251 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:42:14.251 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:42:14.252 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:42:14.260 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: expr 2 + 2 2020-03-11T22:42:14.262 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: seq 1 4 2020-03-11T22:42:14.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:42:14.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_1 td/divergent-priors/existing 2020-03-11T22:42:14.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:42:14.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_2 td/divergent-priors/existing 2020-03-11T22:42:14.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:42:14.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_3 td/divergent-priors/existing 2020-03-11T22:42:14.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:42:14.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_4 td/divergent-priors/existing 2020-03-11T22:42:14.377 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: seq 1 2 2020-03-11T22:42:14.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:42:14.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_1 td/divergent-priors/existing 2020-03-11T22:42:14.403 INFO:tasks.workunit.client.0.smithi145.stderr:error getting test/newobject_1: (2) No such file or directory 2020-03-11T22:42:14.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:42:14.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_2 td/divergent-priors/existing 2020-03-11T22:42:14.431 INFO:tasks.workunit.client.0.smithi145.stderr:error getting test/newobject_2: (2) No such file or directory 2020-03-11T22:42:14.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:602: TEST_divergent_2: rm -f td/divergent-priors/existing 2020-03-11T22:42:14.434 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2020-03-11T22:42:14.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:42:14.438 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 28'106 (28'4) delete 1:70b3bf12:::existing_4:head by client.4539.0:1 2020-03-11T22:40:00.857512+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:42:14.438 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 20 _merge_object_divergent_entries: keeping 28'106 (28'4) delete 1:70b3bf12:::existing_4:head by client.4539.0:1 2020-03-11T22:40:00.857512+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:42:14.438 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:42:14.438 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 28'4 first_divergent_update: 28'106 last_divergent_update: 28'106 2020-03-11T22:42:14.438 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 10 _merge_object_divergent_entries: more recent entry found: 40'101 (28'4) modify 1:70b3bf12:::existing_4:head by client.4652.0:1 2020-03-11T22:40:55.709041+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0, already merged 2020-03-11T22:42:14.438 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 28'103 (0'0) modify 1:75ac5091:::newobject_2:head by client.4532.0:1 2020-03-11T22:40:00.851519+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:42:14.439 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 20 _merge_object_divergent_entries: keeping 28'103 (0'0) modify 1:75ac5091:::newobject_2:head by client.4532.0:1 2020-03-11T22:40:00.851519+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:42:14.439 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:42:14.439 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 28'103 last_divergent_update: 28'103 2020-03-11T22:42:14.439 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2020-03-11T22:42:14.439 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2020-03-11T22:42:14.439 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 28'105 (28'3) delete 1:8881e6d8:::existing_3:head by client.4538.0:1 2020-03-11T22:40:00.855965+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:42:14.439 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 20 _merge_object_divergent_entries: keeping 28'105 (28'3) delete 1:8881e6d8:::existing_3:head by client.4538.0:1 2020-03-11T22:40:00.855965+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:42:14.440 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2020-03-11T22:42:14.440 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 28'3 first_divergent_update: 28'105 last_divergent_update: 28'105 2020-03-11T22:42:14.440 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2020-03-11T22:42:14.440 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:42:14.440 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 28'106 2020-03-11T22:42:14.440 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 28'105 (28'3) delete 1:8881e6d8:::existing_3:head by client.4538.0:1 2020-03-11T22:40:00.855965+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:42:14.440 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2020-03-11T22:42:14.440 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 28'3 <= info.log_tail 28'90 2020-03-11T22:42:14.441 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 28'102 (28'2) modify 1:97db2a0f:::existing_2:head by client.4527.0:1 2020-03-11T22:40:00.844169+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:42:14.441 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 20 _merge_object_divergent_entries: keeping 28'102 (28'2) modify 1:97db2a0f:::existing_2:head by client.4527.0:1 2020-03-11T22:40:00.844169+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:42:14.441 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:42:14.441 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 28'2 first_divergent_update: 28'102 last_divergent_update: 28'102 2020-03-11T22:42:14.441 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2020-03-11T22:42:14.441 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:42:14.441 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 28'106 2020-03-11T22:42:14.442 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 28'102 (28'2) modify 1:97db2a0f:::existing_2:head by client.4527.0:1 2020-03-11T22:40:00.844169+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:42:14.442 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2020-03-11T22:42:14.442 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 28'2 <= info.log_tail 28'90 2020-03-11T22:42:14.442 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 28'101 (28'1) modify 1:d858c035:::existing_1:head by client.4525.0:1 2020-03-11T22:40:00.838780+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:42:14.442 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 20 _merge_object_divergent_entries: keeping 28'101 (28'1) modify 1:d858c035:::existing_1:head by client.4525.0:1 2020-03-11T22:40:00.838780+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:42:14.442 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:42:14.443 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 28'1 first_divergent_update: 28'101 last_divergent_update: 28'101 2020-03-11T22:42:14.443 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2020-03-11T22:42:14.443 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:42:14.444 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 28'106 2020-03-11T22:42:14.444 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 28'101 (28'1) modify 1:d858c035:::existing_1:head by client.4525.0:1 2020-03-11T22:40:00.838780+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:42:14.444 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2020-03-11T22:42:14.444 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 28'1 <= info.log_tail 28'90 2020-03-11T22:42:14.444 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 28'104 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4533.0:1 2020-03-11T22:40:00.852238+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:42:14.444 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 20 _merge_object_divergent_entries: keeping 28'104 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4533.0:1 2020-03-11T22:40:00.852238+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:42:14.444 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:42:14.445 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 28'104 last_divergent_update: 28'104 2020-03-11T22:42:14.445 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2020-03-11T22:42:14.445 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:41:38.596+0000 7fbdb51d3700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2020-03-11T22:42:14.445 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2020-03-11T22:42:14.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:42:14.445 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2020-03-11T22:42:14.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:42:14.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:617: TEST_divergent_2: echo success 2020-03-11T22:42:14.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:619: TEST_divergent_2: rm td/divergent-priors/td/divergent-priors/exp.17402.out 2020-03-11T22:42:14.447 INFO:tasks.workunit.client.0.smithi145.stdout:success 2020-03-11T22:42:14.447 INFO:tasks.workunit.client.0.smithi145.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.17402.out': No such file or directory 2020-03-11T22:42:14.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:621: TEST_divergent_2: delete_pool test 2020-03-11T22:42:14.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test 2020-03-11T22:42:14.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-03-11T22:42:14.917 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' does not exist 2020-03-11T22:42:14.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:622: TEST_divergent_2: kill_daemons td/divergent-priors 2020-03-11T22:42:14.925 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:42:14.925 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:42:14.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:42:14.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:42:14.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:42:15.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:42:15.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2020-03-11T22:42:15.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:42:15.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:42:15.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:42:15.034 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:42:15.034 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:42:15.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:42:15.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:42:15.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:42:15.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:42:15.039 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:42:15.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:42:15.040 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:42:15.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:42:15.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:42:15.041 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:42:15.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:42:15.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:42:15.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:42:15.043 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:42:15.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:42:15.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:42:15.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:42:15.057 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:42:15.057 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:15.057 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:42:15.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17402 2020-03-11T22:42:15.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:42:15.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:42:15.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2020-03-11T22:42:15.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2020-03-11T22:42:15.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2020-03-11T22:42:15.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2020-03-11T22:42:15.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:42:15.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:42:15.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:42:15.060 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:42:15.060 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:42:15.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:42:15.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:42:15.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:42:15.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:42:15.061 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:42:15.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:42:15.062 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:42:15.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:42:15.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:42:15.063 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:42:15.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:42:15.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:42:15.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:42:15.065 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:42:15.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:42:15.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:42:15.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:42:15.067 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:42:15.067 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:15.067 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:42:15.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17402 2020-03-11T22:42:15.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:42:15.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:42:15.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2020-03-11T22:42:15.069 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:42:15.069 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:15.070 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:42:15.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17402 2020-03-11T22:42:15.071 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:42:15.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-03-11T22:42:15.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:42:15.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2020-03-11T22:42:15.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_3 td/divergent-priors 2020-03-11T22:42:15.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:628: TEST_divergent_3: local dir=td/divergent-priors 2020-03-11T22:42:15.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:631: TEST_divergent_3: local dummyfile=/etc/fstab 2020-03-11T22:42:15.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:632: TEST_divergent_3: local dummyfile2=/etc/resolv.conf 2020-03-11T22:42:15.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:634: TEST_divergent_3: local num_osds=3 2020-03-11T22:42:15.073 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: expr 3 - 1 2020-03-11T22:42:15.074 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: seq 0 2 2020-03-11T22:42:15.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: local 'osds=0 2020-03-11T22:42:15.074 INFO:tasks.workunit.client.0.smithi145.stderr:1 2020-03-11T22:42:15.074 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2020-03-11T22:42:15.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:636: TEST_divergent_3: run_mon td/divergent-priors a 2020-03-11T22:42:15.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2020-03-11T22:42:15.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:42:15.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:42:15.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:42:15.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2020-03-11T22:42:15.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2020-03-11T22:42:15.301 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:42:15.301 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:42:15.301 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:42:15.301 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:42:15.302 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:15.302 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:42:15.302 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:42:15.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:42:15.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:42:15.491 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:42:15.491 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:42:15.492 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:42:15.492 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:42:15.493 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:42:15.493 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:42:15.493 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:42:15.493 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:42:15.494 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:42:15.494 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:15.494 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:42:15.494 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17402/ceph-mon.a.asok 2020-03-11T22:42:15.494 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:42:15.494 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17402/ceph-mon.a.asok config get fsid 2020-03-11T22:42:15.580 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:42:15.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:42:15.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:42:15.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:42:15.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:42:15.581 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:42:15.581 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:42:15.582 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:42:15.582 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:42:15.582 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:15.582 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:42:15.582 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17402/ceph-mon.a.asok 2020-03-11T22:42:15.583 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:42:15.583 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17402/ceph-mon.a.asok config get mon_host 2020-03-11T22:42:15.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:637: TEST_divergent_3: run_mgr td/divergent-priors x 2020-03-11T22:42:15.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/divergent-priors 2020-03-11T22:42:15.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:42:15.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:42:15.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:42:15.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/divergent-priors/x 2020-03-11T22:42:15.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:42:16.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:42:16.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:42:16.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:42:16.131 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:42:16.131 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:16.131 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:42:16.132 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:42:16.132 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:42:16.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:42:16.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2020-03-11T22:42:16.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 0 2020-03-11T22:42:16.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:42:16.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:42:16.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:42:16.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:42:16.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:42:16.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:42:16.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:42:16.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:42:16.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:42:16.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:42:16.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:42:16.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:42:16.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:42:16.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:42:16.162 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:42:16.162 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:42:16.162 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:42:16.162 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:42:16.162 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:16.163 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:42:16.163 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:42:16.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:42:16.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:42:16.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:42:16.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:42:16.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:42:16.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:42:16.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:42:16.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:42:16.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:42:16.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:42:16.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:42:16.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:42:16.165 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:42:16.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=25919b39-e67c-442a-a46f-e5d1e5ebdbd6 2020-03-11T22:42:16.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 25919b39-e67c-442a-a46f-e5d1e5ebdbd6' 2020-03-11T22:42:16.166 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:42:16.166 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 25919b39-e67c-442a-a46f-e5d1e5ebdbd6 2020-03-11T22:42:16.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBIaWleYZF/ChAAwe9HV23c6uT2SXC4DTlOXg== 2020-03-11T22:42:16.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBIaWleYZF/ChAAwe9HV23c6uT2SXC4DTlOXg=="}' 2020-03-11T22:42:16.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 25919b39-e67c-442a-a46f-e5d1e5ebdbd6 -i td/divergent-priors/0/new.json 2020-03-11T22:42:16.480 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:42:16.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/0/new.json 2020-03-11T22:42:16.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBIaWleYZF/ChAAwe9HV23c6uT2SXC4DTlOXg== --osd-uuid 25919b39-e67c-442a-a46f-e5d1e5ebdbd6 2020-03-11T22:42:16.518 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:42:16.516+0000 7efdc2973ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:16.520 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:42:16.518+0000 7efdc2973ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:16.522 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:42:16.520+0000 7efdc2973ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:16.523 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:42:16.520+0000 7efdc2973ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:42:16.523 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:42:16.520+0000 7efdc2973ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:42:16.523 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:42:16.520+0000 7efdc2973ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:42:16.566 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:42:16.564+0000 7efdc2973ec0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2020-03-11T22:42:19.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/0/keyring 2020-03-11T22:42:19.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:42:19.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:42:19.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:42:19.556 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2020-03-11T22:42:20.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:42:20.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:42:20.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:42:20.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:42:20.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:42:20.006 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2020-03-11T22:42:20.031 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:42:20.028+0000 7f79877c1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:20.032 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:42:20.030+0000 7f79877c1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:20.035 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:42:20.032+0000 7f79877c1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:20.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:42:20.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:42:20.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:42:20.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:42:20.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:42:20.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:42:20.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:42:20.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:42:20.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:42:20.370 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:42:20.555 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:42:20.552+0000 7f79877c1ec0 -1 Falling back to public interface 2020-03-11T22:42:20.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:42:21.258 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:42:21.255+0000 7f79877c1ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:42:21.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:42:21.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:42:21.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:42:21.721 INFO:tasks.workunit.client.0.smithi145.stdout:1 2020-03-11T22:42:21.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:42:21.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:42:22.108 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/34473,v1:127.0.0.1:6803/34473] [v2:127.0.0.1:6804/34473,v1:127.0.0.1:6805/34473] exists,up 25919b39-e67c-442a-a46f-e5d1e5ebdbd6 2020-03-11T22:42:22.108 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 72907ff7-85ea-4a4a-b62c-397e4c805acb 2020-03-11T22:42:22.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:42:22.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:42:22.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:42:22.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2020-03-11T22:42:22.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 1 2020-03-11T22:42:22.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:42:22.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:42:22.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:42:22.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:42:22.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:42:22.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:42:22.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:42:22.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:42:22.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:42:22.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:42:22.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:42:22.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:42:22.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:42:22.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:42:22.112 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:42:22.112 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:42:22.112 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:42:22.112 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:42:22.113 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:22.113 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:42:22.113 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:42:22.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:42:22.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:42:22.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:42:22.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:42:22.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:42:22.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:42:22.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:42:22.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:42:22.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:42:22.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:42:22.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:42:22.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:42:22.115 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:42:22.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=72907ff7-85ea-4a4a-b62c-397e4c805acb 2020-03-11T22:42:22.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 72907ff7-85ea-4a4a-b62c-397e4c805acb' 2020-03-11T22:42:22.115 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:42:22.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBOaWlefAFfBxAA9ZkKL7adGA4lQKGWTbUrIQ== 2020-03-11T22:42:22.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBOaWlefAFfBxAA9ZkKL7adGA4lQKGWTbUrIQ=="}' 2020-03-11T22:42:22.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 72907ff7-85ea-4a4a-b62c-397e4c805acb -i td/divergent-priors/1/new.json 2020-03-11T22:42:22.586 INFO:tasks.workunit.client.0.smithi145.stdout:1 2020-03-11T22:42:22.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/1/new.json 2020-03-11T22:42:22.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBOaWlefAFfBxAA9ZkKL7adGA4lQKGWTbUrIQ== --osd-uuid 72907ff7-85ea-4a4a-b62c-397e4c805acb 2020-03-11T22:42:22.626 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:42:22.622+0000 7f96c7ceaec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:22.626 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:42:22.623+0000 7f96c7ceaec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:22.627 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:42:22.625+0000 7f96c7ceaec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:22.628 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:42:22.625+0000 7f96c7ceaec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:42:22.628 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:42:22.625+0000 7f96c7ceaec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:42:22.628 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:42:22.625+0000 7f96c7ceaec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:42:22.698 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:42:22.695+0000 7f96c7ceaec0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2020-03-11T22:42:25.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/1/keyring 2020-03-11T22:42:25.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:42:25.657 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2020-03-11T22:42:25.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:42:25.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:42:26.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:42:26.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:42:26.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:42:26.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:42:26.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:42:26.114 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2020-03-11T22:42:26.139 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:42:26.136+0000 7f9d2a3d9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:26.141 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:42:26.138+0000 7f9d2a3d9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:26.143 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:42:26.140+0000 7f9d2a3d9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:26.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:42:26.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:42:26.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:42:26.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:42:26.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:42:26.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:42:26.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:42:26.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:42:26.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:42:26.469 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:42:26.659 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:42:26.656+0000 7f9d2a3d9ec0 -1 Falling back to public interface 2020-03-11T22:42:26.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:42:27.421 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:42:27.419+0000 7f9d2a3d9ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:42:27.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:42:27.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:42:27.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:42:27.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:42:27.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:42:27.812 INFO:tasks.workunit.client.0.smithi145.stdout:1 2020-03-11T22:42:28.166 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/34828,v1:127.0.0.1:6811/34828] [v2:127.0.0.1:6812/34828,v1:127.0.0.1:6813/34828] exists,up 72907ff7-85ea-4a4a-b62c-397e4c805acb 2020-03-11T22:42:28.167 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 f1f86676-d3cf-4822-8bec-ab32748439d8 2020-03-11T22:42:28.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:42:28.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:42:28.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:42:28.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2020-03-11T22:42:28.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 2 2020-03-11T22:42:28.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:42:28.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:42:28.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:42:28.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:42:28.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:42:28.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:42:28.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:42:28.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:42:28.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:42:28.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:42:28.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:42:28.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:42:28.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:42:28.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:42:28.170 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:42:28.170 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:42:28.170 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:42:28.170 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:42:28.170 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:28.170 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:42:28.171 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:42:28.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:42:28.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:42:28.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:42:28.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:42:28.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:42:28.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:42:28.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:42:28.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:42:28.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:42:28.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:42:28.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:42:28.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:42:28.172 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:42:28.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=f1f86676-d3cf-4822-8bec-ab32748439d8 2020-03-11T22:42:28.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 f1f86676-d3cf-4822-8bec-ab32748439d8' 2020-03-11T22:42:28.173 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:42:28.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBUaWleBybeChAAxlg2/ysx8a30VkhDT+9YEQ== 2020-03-11T22:42:28.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBUaWleBybeChAAxlg2/ysx8a30VkhDT+9YEQ=="}' 2020-03-11T22:42:28.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new f1f86676-d3cf-4822-8bec-ab32748439d8 -i td/divergent-priors/2/new.json 2020-03-11T22:42:28.679 INFO:tasks.workunit.client.0.smithi145.stdout:2 2020-03-11T22:42:28.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/2/new.json 2020-03-11T22:42:28.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBUaWleBybeChAAxlg2/ysx8a30VkhDT+9YEQ== --osd-uuid f1f86676-d3cf-4822-8bec-ab32748439d8 2020-03-11T22:42:28.718 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:42:28.715+0000 7f152246cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:28.720 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:42:28.717+0000 7f152246cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:28.721 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:42:28.718+0000 7f152246cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:28.722 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:42:28.719+0000 7f152246cec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:42:28.722 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:42:28.719+0000 7f152246cec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:42:28.722 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:42:28.719+0000 7f152246cec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:42:28.804 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:42:28.801+0000 7f152246cec0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2020-03-11T22:42:31.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/2/keyring 2020-03-11T22:42:31.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:42:31.756 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2020-03-11T22:42:31.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:42:31.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:42:32.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:42:32.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:42:32.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:42:32.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:42:32.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:42:32.215 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2020-03-11T22:42:32.240 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:42:32.238+0000 7f551a366ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:32.242 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:42:32.240+0000 7f551a366ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:32.245 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:42:32.241+0000 7f551a366ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:32.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:42:32.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:42:32.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:42:32.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:42:32.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:42:32.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:42:32.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:42:32.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:42:32.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:42:32.572 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:42:32.760 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:42:32.757+0000 7f551a366ec0 -1 Falling back to public interface 2020-03-11T22:42:32.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:42:33.513 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:42:33.509+0000 7f551a366ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:42:33.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:42:33.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:42:33.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:42:33.921 INFO:tasks.workunit.client.0.smithi145.stdout:1 2020-03-11T22:42:33.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:42:33.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:42:34.296 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/35183,v1:127.0.0.1:6819/35183] [v2:127.0.0.1:6820/35183,v1:127.0.0.1:6821/35183] exists,up f1f86676-d3cf-4822-8bec-ab32748439d8 2020-03-11T22:42:34.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:42:34.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:42:34.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:42:34.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:643: TEST_divergent_3: ceph osd set noout 2020-03-11T22:42:34.830 INFO:tasks.workunit.client.0.smithi145.stderr:noout is set 2020-03-11T22:42:34.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:644: TEST_divergent_3: ceph osd set noin 2020-03-11T22:42:35.440 INFO:tasks.workunit.client.0.smithi145.stderr:noin is set 2020-03-11T22:42:35.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:645: TEST_divergent_3: ceph osd set nodown 2020-03-11T22:42:36.175 INFO:tasks.workunit.client.0.smithi145.stderr:nodown is set 2020-03-11T22:42:36.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:646: TEST_divergent_3: create_pool test 1 1 2020-03-11T22:42:36.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-03-11T22:42:36.775 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' already exists 2020-03-11T22:42:36.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:42:37.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:647: TEST_divergent_3: ceph osd pool set test size 3 2020-03-11T22:42:38.395 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 3 2020-03-11T22:42:38.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:648: TEST_divergent_3: ceph osd pool set test min_size 2 2020-03-11T22:42:39.118 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 min_size to 2 2020-03-11T22:42:39.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:651: TEST_divergent_3: ceph osd pool set test pg_autoscale_mode on 2020-03-11T22:42:39.765 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 pg_autoscale_mode to on 2020-03-11T22:42:39.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:653: TEST_divergent_3: flush_pg_stats 2020-03-11T22:42:39.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:42:39.774 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:42:40.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:42:40.118 INFO:tasks.workunit.client.0.smithi145.stderr:1 2020-03-11T22:42:40.118 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2020-03-11T22:42:40.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:42:40.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:40.118 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:42:40.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836486 2020-03-11T22:42:40.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486' 2020-03-11T22:42:40.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:40.246 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:42:40.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672965 2020-03-11T22:42:40.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672965' 2020-03-11T22:42:40.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:40.376 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:42:40.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509445 2020-03-11T22:42:40.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672965 2-64424509445' 2020-03-11T22:42:40.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:40.505 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836486 2020-03-11T22:42:40.505 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:40.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:42:40.506 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836486 2020-03-11T22:42:40.506 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:40.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836486 2020-03-11T22:42:40.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2020-03-11T22:42:40.507 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:42:40.507 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836486 2020-03-11T22:42:40.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836484 -lt 21474836486 2020-03-11T22:42:40.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:42:41.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:42:41.883 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:42:42.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836486 -lt 21474836486 2020-03-11T22:42:42.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:42.234 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672965 2020-03-11T22:42:42.235 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:42.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:42:42.236 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672965 2020-03-11T22:42:42.236 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:42.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672965 2020-03-11T22:42:42.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2020-03-11T22:42:42.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:42:42.237 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672965 2020-03-11T22:42:42.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672965 -lt 42949672965 2020-03-11T22:42:42.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:42.580 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509445 2020-03-11T22:42:42.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:42.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:42:42.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509445 2020-03-11T22:42:42.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:42.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509445 2020-03-11T22:42:42.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2020-03-11T22:42:42.583 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:42:42.583 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509445 2020-03-11T22:42:42.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509445 -lt 64424509445 2020-03-11T22:42:42.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: wait_for_clean 2020-03-11T22:42:42.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:42:42.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:42:42.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:42:42.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:42:42.937 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:42:42.937 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:42:42.937 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:42:42.937 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:42:42.937 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:42:42.938 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:42:43.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:42:43.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:42:43.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:42:43.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:42:43.029 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:42:43.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:42:43.370 INFO:tasks.workunit.client.0.smithi145.stderr:1 2020-03-11T22:42:43.370 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2020-03-11T22:42:43.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:42:43.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:43.371 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:42:43.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836489 2020-03-11T22:42:43.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489' 2020-03-11T22:42:43.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:43.501 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:42:43.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672967 2020-03-11T22:42:43.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-42949672967' 2020-03-11T22:42:43.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:43.631 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:42:43.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509447 2020-03-11T22:42:43.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-42949672967 2-64424509447' 2020-03-11T22:42:43.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:43.760 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836489 2020-03-11T22:42:43.760 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:43.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:42:43.761 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836489 2020-03-11T22:42:43.761 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:43.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836489 2020-03-11T22:42:43.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2020-03-11T22:42:43.762 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:42:43.763 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836489 2020-03-11T22:42:44.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836487 -lt 21474836489 2020-03-11T22:42:44.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:42:45.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:42:45.106 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:42:45.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836487 -lt 21474836489 2020-03-11T22:42:45.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:42:46.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 299 -eq 0 ']' 2020-03-11T22:42:46.486 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:42:46.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836489 -lt 21474836489 2020-03-11T22:42:46.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:46.845 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672967 2020-03-11T22:42:46.845 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:46.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:42:46.846 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672967 2020-03-11T22:42:46.846 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:46.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672967 2020-03-11T22:42:46.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2020-03-11T22:42:46.847 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:42:46.847 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672967 2020-03-11T22:42:47.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672968 -lt 42949672967 2020-03-11T22:42:47.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:47.210 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509447 2020-03-11T22:42:47.210 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:47.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:42:47.211 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509447 2020-03-11T22:42:47.211 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:47.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509447 2020-03-11T22:42:47.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2020-03-11T22:42:47.212 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:42:47.213 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509447 2020-03-11T22:42:47.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509448 -lt 64424509447 2020-03-11T22:42:47.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:42:47.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:47.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:48.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:42:48.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:42:48.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:42:48.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:42:48.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:42:48.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:42:48.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:42:48.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:42:48.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:42:48.353 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:42:48.353 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:48.353 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:48.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:42:48.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:42:48.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:42:48.797 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: ceph pg dump pgs --format=json 2020-03-11T22:42:48.797 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: jq '.pg_stats[0].up_primary' 2020-03-11T22:42:49.155 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:42:49.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: local divergent=1 2020-03-11T22:42:49.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:658: TEST_divergent_3: echo 'primary and soon to be divergent is 1' 2020-03-11T22:42:49.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:659: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:42:49.164 INFO:tasks.workunit.client.0.smithi145.stdout:primary and soon to be divergent is 1 2020-03-11T22:42:49.483 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:42:49.483 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:42:38.574930+0000 0'0 28:13 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:42:49.483 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:42:49.483 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:42:49.485 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:42:49.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:660: TEST_divergent_3: local non_divergent= 2020-03-11T22:42:49.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2020-03-11T22:42:49.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 0 = 1 ']' 2020-03-11T22:42:49.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:666: TEST_divergent_3: non_divergent=' 0' 2020-03-11T22:42:49.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2020-03-11T22:42:49.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 1 = 1 ']' 2020-03-11T22:42:49.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:664: TEST_divergent_3: continue 2020-03-11T22:42:49.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2020-03-11T22:42:49.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 2 = 1 ']' 2020-03-11T22:42:49.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:666: TEST_divergent_3: non_divergent=' 0 2' 2020-03-11T22:42:49.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:669: TEST_divergent_3: echo 'writing initial objects' 2020-03-11T22:42:49.496 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: seq 1 100 2020-03-11T22:42:49.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:49.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_1 /etc/fstab 2020-03-11T22:42:49.497 INFO:tasks.workunit.client.0.smithi145.stdout:writing initial objects 2020-03-11T22:42:49.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:49.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_2 /etc/fstab 2020-03-11T22:42:49.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:49.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_3 /etc/fstab 2020-03-11T22:42:49.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:49.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_4 /etc/fstab 2020-03-11T22:42:49.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:49.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_5 /etc/fstab 2020-03-11T22:42:49.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:49.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_6 /etc/fstab 2020-03-11T22:42:49.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:49.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_7 /etc/fstab 2020-03-11T22:42:49.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:49.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_8 /etc/fstab 2020-03-11T22:42:49.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:49.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_9 /etc/fstab 2020-03-11T22:42:49.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:49.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_10 /etc/fstab 2020-03-11T22:42:50.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:50.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_11 /etc/fstab 2020-03-11T22:42:50.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:50.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_12 /etc/fstab 2020-03-11T22:42:50.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:50.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_13 /etc/fstab 2020-03-11T22:42:50.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:50.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_14 /etc/fstab 2020-03-11T22:42:50.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:50.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_15 /etc/fstab 2020-03-11T22:42:50.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:50.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_16 /etc/fstab 2020-03-11T22:42:50.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:50.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_17 /etc/fstab 2020-03-11T22:42:50.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:50.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_18 /etc/fstab 2020-03-11T22:42:50.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:50.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_19 /etc/fstab 2020-03-11T22:42:50.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:50.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_20 /etc/fstab 2020-03-11T22:42:50.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:50.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_21 /etc/fstab 2020-03-11T22:42:50.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:50.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_22 /etc/fstab 2020-03-11T22:42:50.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:50.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_23 /etc/fstab 2020-03-11T22:42:50.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:50.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_24 /etc/fstab 2020-03-11T22:42:50.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:50.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_25 /etc/fstab 2020-03-11T22:42:50.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:50.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_26 /etc/fstab 2020-03-11T22:42:50.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:50.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_27 /etc/fstab 2020-03-11T22:42:50.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:50.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_28 /etc/fstab 2020-03-11T22:42:50.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:50.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_29 /etc/fstab 2020-03-11T22:42:51.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:51.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_30 /etc/fstab 2020-03-11T22:42:51.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:51.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_31 /etc/fstab 2020-03-11T22:42:51.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:51.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_32 /etc/fstab 2020-03-11T22:42:51.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:51.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_33 /etc/fstab 2020-03-11T22:42:51.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:51.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_34 /etc/fstab 2020-03-11T22:42:51.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:51.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_35 /etc/fstab 2020-03-11T22:42:51.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:51.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_36 /etc/fstab 2020-03-11T22:42:51.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:51.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_37 /etc/fstab 2020-03-11T22:42:51.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:51.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_38 /etc/fstab 2020-03-11T22:42:51.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:51.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_39 /etc/fstab 2020-03-11T22:42:51.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:51.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_40 /etc/fstab 2020-03-11T22:42:51.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:51.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_41 /etc/fstab 2020-03-11T22:42:51.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:51.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_42 /etc/fstab 2020-03-11T22:42:51.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:51.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_43 /etc/fstab 2020-03-11T22:42:51.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:51.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_44 /etc/fstab 2020-03-11T22:42:51.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:51.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_45 /etc/fstab 2020-03-11T22:42:52.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_46 /etc/fstab 2020-03-11T22:42:52.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_47 /etc/fstab 2020-03-11T22:42:52.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_48 /etc/fstab 2020-03-11T22:42:52.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_49 /etc/fstab 2020-03-11T22:42:52.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_50 /etc/fstab 2020-03-11T22:42:52.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_51 /etc/fstab 2020-03-11T22:42:52.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_52 /etc/fstab 2020-03-11T22:42:52.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_53 /etc/fstab 2020-03-11T22:42:52.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_54 /etc/fstab 2020-03-11T22:42:52.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_55 /etc/fstab 2020-03-11T22:42:52.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_56 /etc/fstab 2020-03-11T22:42:52.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_57 /etc/fstab 2020-03-11T22:42:52.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_58 /etc/fstab 2020-03-11T22:42:52.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_59 /etc/fstab 2020-03-11T22:42:52.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_60 /etc/fstab 2020-03-11T22:42:52.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_61 /etc/fstab 2020-03-11T22:42:52.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_62 /etc/fstab 2020-03-11T22:42:52.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_63 /etc/fstab 2020-03-11T22:42:52.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:52.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_64 /etc/fstab 2020-03-11T22:42:53.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_65 /etc/fstab 2020-03-11T22:42:53.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_66 /etc/fstab 2020-03-11T22:42:53.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_67 /etc/fstab 2020-03-11T22:42:53.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_68 /etc/fstab 2020-03-11T22:42:53.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_69 /etc/fstab 2020-03-11T22:42:53.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_70 /etc/fstab 2020-03-11T22:42:53.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_71 /etc/fstab 2020-03-11T22:42:53.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_72 /etc/fstab 2020-03-11T22:42:53.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_73 /etc/fstab 2020-03-11T22:42:53.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_74 /etc/fstab 2020-03-11T22:42:53.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_75 /etc/fstab 2020-03-11T22:42:53.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_76 /etc/fstab 2020-03-11T22:42:53.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_77 /etc/fstab 2020-03-11T22:42:53.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_78 /etc/fstab 2020-03-11T22:42:53.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_79 /etc/fstab 2020-03-11T22:42:53.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_80 /etc/fstab 2020-03-11T22:42:53.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_81 /etc/fstab 2020-03-11T22:42:53.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_82 /etc/fstab 2020-03-11T22:42:53.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:53.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_83 /etc/fstab 2020-03-11T22:42:54.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_84 /etc/fstab 2020-03-11T22:42:54.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_85 /etc/fstab 2020-03-11T22:42:54.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_86 /etc/fstab 2020-03-11T22:42:54.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_87 /etc/fstab 2020-03-11T22:42:54.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_88 /etc/fstab 2020-03-11T22:42:54.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_89 /etc/fstab 2020-03-11T22:42:54.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_90 /etc/fstab 2020-03-11T22:42:54.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_91 /etc/fstab 2020-03-11T22:42:54.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_92 /etc/fstab 2020-03-11T22:42:54.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_93 /etc/fstab 2020-03-11T22:42:54.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_94 /etc/fstab 2020-03-11T22:42:54.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_95 /etc/fstab 2020-03-11T22:42:54.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_96 /etc/fstab 2020-03-11T22:42:54.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_97 /etc/fstab 2020-03-11T22:42:54.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_98 /etc/fstab 2020-03-11T22:42:54.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:54.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_99 /etc/fstab 2020-03-11T22:42:55.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:42:55.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_100 /etc/fstab 2020-03-11T22:42:55.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:42:55.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: wait_for_clean 2020-03-11T22:42:55.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:42:55.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:42:55.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:42:55.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:42:55.117 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:42:55.117 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:42:55.118 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:42:55.118 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:42:55.118 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:42:55.118 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:42:55.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:42:55.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:42:55.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:42:55.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:42:55.174 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:42:55.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:42:55.558 INFO:tasks.workunit.client.0.smithi145.stderr:1 2020-03-11T22:42:55.558 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2020-03-11T22:42:55.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:42:55.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:55.558 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:42:55.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836493 2020-03-11T22:42:55.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493' 2020-03-11T22:42:55.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:55.687 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:42:55.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672972 2020-03-11T22:42:55.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672972' 2020-03-11T22:42:55.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:55.818 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:42:55.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509452 2020-03-11T22:42:55.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672972 2-64424509452' 2020-03-11T22:42:55.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:55.947 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836493 2020-03-11T22:42:55.947 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:55.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:42:55.947 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836493 2020-03-11T22:42:55.948 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:55.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836493 2020-03-11T22:42:55.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2020-03-11T22:42:55.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:42:55.949 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836493 2020-03-11T22:42:56.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836491 -lt 21474836493 2020-03-11T22:42:56.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:42:57.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:42:57.292 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:42:57.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836493 -lt 21474836493 2020-03-11T22:42:57.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:57.667 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672972 2020-03-11T22:42:57.668 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:57.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:42:57.668 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672972 2020-03-11T22:42:57.669 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:57.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672972 2020-03-11T22:42:57.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2020-03-11T22:42:57.670 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:42:57.670 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672972 2020-03-11T22:42:58.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672972 -lt 42949672972 2020-03-11T22:42:58.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:58.013 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509452 2020-03-11T22:42:58.013 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:58.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:42:58.015 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509452 2020-03-11T22:42:58.015 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:58.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509452 2020-03-11T22:42:58.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2020-03-11T22:42:58.016 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:42:58.016 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509452 2020-03-11T22:42:58.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509452 -lt 64424509452 2020-03-11T22:42:58.376 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:42:58.376 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:58.376 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:58.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:42:58.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:42:58.827 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:42:58.827 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:42:58.828 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:42:58.828 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:42:58.828 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:42:58.828 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:42:59.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:42:59.161 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:42:59.161 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:59.162 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:59.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:42:59.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:42:59.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:42:59.606 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: get_pg test existing_1 2020-03-11T22:42:59.607 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local poolname=test 2020-03-11T22:42:59.607 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local objectname=existing_1 2020-03-11T22:42:59.607 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: ceph --format json osd map test existing_1 2020-03-11T22:42:59.607 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: jq -r .pgid 2020-03-11T22:42:59.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: local pgid=1.0 2020-03-11T22:42:59.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: echo 'blackholing osds 0 2' 2020-03-11T22:42:59.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:682: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:42:59.951 INFO:tasks.workunit.client.0.smithi145.stdout:blackholing osds 0 2 2020-03-11T22:43:00.271 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:43:00.271 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2020-03-11T22:42:38.574930+0000 30'100 30:113 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:43:00.271 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:43:00.271 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:43:00.273 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:43:00.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:683: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:43:00.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: get_asok_path osd.0 2020-03-11T22:43:00.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:43:00.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:43:00.282 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:43:00.282 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:00.282 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:43:00.283 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17402/ceph-osd.0.asok 2020-03-11T22:43:00.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:43:00.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17402/ceph-osd.0.asok config set objectstore_blackhole 1 2020-03-11T22:43:00.367 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:43:00.367 INFO:tasks.workunit.client.0.smithi145.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:43:00.368 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:43:00.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:683: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:43:00.375 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: get_asok_path osd.2 2020-03-11T22:43:00.375 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:43:00.376 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:43:00.376 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:43:00.376 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:00.376 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:43:00.376 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17402/ceph-osd.2.asok 2020-03-11T22:43:00.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:43:00.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17402/ceph-osd.2.asok config set objectstore_blackhole 1 2020-03-11T22:43:00.462 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:43:00.462 INFO:tasks.workunit.client.0.smithi145.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:43:00.462 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:43:00.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:689: TEST_divergent_3: echo 'create new divergent objects' 2020-03-11T22:43:00.469 INFO:tasks.workunit.client.0.smithi145.stdout:create new divergent objects 2020-03-11T22:43:00.469 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: seq 1 2 2020-03-11T22:43:00.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:43:00.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:43:00.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:692: TEST_divergent_3: rados -p test create newobject_1 2020-03-11T22:43:00.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:695: TEST_divergent_3: echo 'writing divergent objects' 2020-03-11T22:43:00.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:692: TEST_divergent_3: rados -p test create newobject_2 2020-03-11T22:43:00.471 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: seq 1 2 2020-03-11T22:43:00.471 INFO:tasks.workunit.client.0.smithi145.stdout:writing divergent objects 2020-03-11T22:43:00.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2020-03-11T22:43:00.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2020-03-11T22:43:00.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: rados -p test put existing_1 /etc/resolv.conf 2020-03-11T22:43:00.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:701: TEST_divergent_3: echo 'remove divergent objects' 2020-03-11T22:43:00.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: rados -p test put existing_2 /etc/resolv.conf 2020-03-11T22:43:00.473 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: seq 1 2 2020-03-11T22:43:00.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-03-11T22:43:00.474 INFO:tasks.workunit.client.0.smithi145.stdout:remove divergent objects 2020-03-11T22:43:00.474 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: expr 1 + 2 2020-03-11T22:43:00.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: rmi=3 2020-03-11T22:43:00.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-03-11T22:43:00.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: rados -p test rm existing_3 2020-03-11T22:43:00.475 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: expr 2 + 2 2020-03-11T22:43:00.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: rmi=4 2020-03-11T22:43:00.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: sleep 10 2020-03-11T22:43:00.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: rados -p test rm existing_4 2020-03-11T22:43:10.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:708: TEST_divergent_3: killall -9 rados 2020-03-11T22:43:10.482 INFO:tasks.workunit.client.0.smithi145.stderr:rados: no process found 2020-03-11T22:43:10.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: echo 'killing all the osds' 2020-03-11T22:43:10.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:712: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:43:10.483 INFO:tasks.workunit.client.0.smithi145.stdout:killing all the osds 2020-03-11T22:43:10.812 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:43:10.812 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 active+clean 2020-03-11T22:42:38.574930+0000 30'106 30:119 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:43:10.812 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:43:10.813 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:43:10.814 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:43:10.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd 2020-03-11T22:43:10.823 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:43:10.823 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:43:10.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:43:10.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:43:10.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:43:10.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:43:10.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2020-03-11T22:43:10.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.0 2020-03-11T22:43:11.422 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already down. 2020-03-11T22:43:11.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2020-03-11T22:43:11.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.1 2020-03-11T22:43:11.938 INFO:tasks.workunit.client.0.smithi145.stderr:osd.1 is already down. 2020-03-11T22:43:11.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2020-03-11T22:43:11.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.2 2020-03-11T22:43:12.431 INFO:tasks.workunit.client.0.smithi145.stderr:osd.2 is already down. 2020-03-11T22:43:12.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:718: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:43:12.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: ceph osd out osd.0 2020-03-11T22:43:12.925 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already out. 2020-03-11T22:43:12.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:718: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:43:12.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: ceph osd out osd.2 2020-03-11T22:43:13.408 INFO:tasks.workunit.client.0.smithi145.stderr:osd.2 is already out. 2020-03-11T22:43:13.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:724: TEST_divergent_3: echo 'bringing up non_divergent 0 2' 2020-03-11T22:43:13.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:725: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:43:13.417 INFO:tasks.workunit.client.0.smithi145.stdout:bringing up non_divergent 0 2 2020-03-11T22:43:13.736 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:43:13.736 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 stale+active+clean 2020-03-11T22:42:38.574930+0000 30'106 30:119 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:43:13.737 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:43:13.737 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:43:13.738 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:43:13.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:43:13.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: activate_osd td/divergent-priors 0 2020-03-11T22:43:13.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:43:13.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:43:13.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:43:13.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:43:13.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:43:13.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:43:13.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:43:13.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:43:13.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:43:13.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:43:13.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:43:13.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:43:13.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:43:13.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:43:13.751 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:43:13.752 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:43:13.752 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:43:13.752 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:43:13.752 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:13.753 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:43:13.753 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:43:13.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:43:13.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:43:13.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:43:13.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:43:13.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:43:13.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:43:13.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:13.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:43:13.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:43:13.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:43:13.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:43:13.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:13.755 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-03-11T22:43:13.755 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2020-03-11T22:43:13.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:43:13.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:43:13.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:43:13.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:43:13.775 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:43:13.773+0000 7f3259318ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:13.777 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:43:13.774+0000 7f3259318ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:13.778 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:43:13.775+0000 7f3259318ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:14.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:43:14.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:43:14.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:43:14.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:43:14.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:43:14.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:14.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:43:14.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:14.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:43:14.097 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:43:14.292 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:43:14.290+0000 7f3259318ec0 -1 Falling back to public interface 2020-03-11T22:43:14.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:43:15.141 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:43:15.138+0000 7f3259318ec0 -1 osd.0 30 log_to_monitors {default=true} 2020-03-11T22:43:15.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:43:15.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:15.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:43:15.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:15.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:43:15.465 INFO:tasks.workunit.client.0.smithi145.stdout:1 2020-03-11T22:43:15.828 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 31 last_clean_interval [5,30) [v2:127.0.0.1:6802/39271,v1:127.0.0.1:6803/39271] [v2:127.0.0.1:6804/39271,v1:127.0.0.1:6805/39271] exists,up 25919b39-e67c-442a-a46f-e5d1e5ebdbd6 2020-03-11T22:43:15.828 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2020-03-11T22:43:15.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:43:15.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:43:15.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:43:15.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:43:15.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: activate_osd td/divergent-priors 2 2020-03-11T22:43:15.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:43:15.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:43:15.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:43:15.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:43:15.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:43:15.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:43:15.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:43:15.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:43:15.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:43:15.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:43:15.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:43:15.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:43:15.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:43:15.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:43:15.831 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:43:15.832 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:43:15.832 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:43:15.832 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:43:15.832 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:15.832 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:43:15.832 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:43:15.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:43:15.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:43:15.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:43:15.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:43:15.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:43:15.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:43:15.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:15.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:43:15.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:43:15.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:43:15.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:43:15.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:15.834 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-03-11T22:43:15.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:43:15.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:43:15.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:43:15.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:43:15.855 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:43:15.852+0000 7fbe2a702ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:15.857 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:43:15.854+0000 7fbe2a702ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:15.858 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:43:15.856+0000 7fbe2a702ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:16.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:43:16.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:43:16.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:43:16.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:43:16.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:43:16.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:16.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:43:16.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:16.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:43:16.208 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:43:16.383 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:43:16.381+0000 7fbe2a702ec0 -1 Falling back to public interface 2020-03-11T22:43:16.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:43:17.224 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:43:17.221+0000 7fbe2a702ec0 -1 osd.2 30 log_to_monitors {default=true} 2020-03-11T22:43:17.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:43:17.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:17.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:43:17.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:17.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:43:17.556 INFO:tasks.workunit.client.0.smithi145.stdout:1 2020-03-11T22:43:17.905 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up out weight 0 up_from 37 up_thru 0 down_at 33 last_clean_interval [15,30) [v2:127.0.0.1:6810/39465,v1:127.0.0.1:6811/39465] [v2:127.0.0.1:6812/39465,v1:127.0.0.1:6813/39465] exists,up f1f86676-d3cf-4822-8bec-ab32748439d8 2020-03-11T22:43:17.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:43:17.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:43:17.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:43:17.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:730: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:43:17.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:732: TEST_divergent_3: ceph osd in osd.0 2020-03-11T22:43:18.431 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already in. 2020-03-11T22:43:18.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:730: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:43:18.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:732: TEST_divergent_3: ceph osd in osd.2 2020-03-11T22:43:19.006 INFO:tasks.workunit.client.0.smithi145.stderr:osd.2 is already in. 2020-03-11T22:43:19.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:43:19.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: wait_for_clean 2020-03-11T22:43:19.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:43:19.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:43:19.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:43:19.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:43:19.015 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:43:19.016 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:43:19.016 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:43:19.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:43:19.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:43:19.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:43:19.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:43:19.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:43:19.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:43:19.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:43:19.074 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:43:19.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:43:19.421 INFO:tasks.workunit.client.0.smithi145.stderr:1 2020-03-11T22:43:19.422 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2020-03-11T22:43:19.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:43:19.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:43:19.422 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:43:19.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=154618822659 2020-03-11T22:43:19.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-154618822659' 2020-03-11T22:43:19.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:43:19.555 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:43:19.645 INFO:tasks.workunit.client.0.smithi145.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:43:19.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:43:19.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-154618822659 1-' 2020-03-11T22:43:19.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:43:19.654 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:43:19.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=158913789955 2020-03-11T22:43:19.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-154618822659 1- 2-158913789955' 2020-03-11T22:43:19.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:43:19.784 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-154618822659 2020-03-11T22:43:19.784 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:43:19.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:43:19.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-154618822659 2020-03-11T22:43:19.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:43:19.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=154618822659 2020-03-11T22:43:19.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 154618822659' 2020-03-11T22:43:19.786 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:43:19.787 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 154618822659 2020-03-11T22:43:20.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 154618822659 2020-03-11T22:43:20.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:43:21.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:43:21.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:43:21.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 154618822659 2020-03-11T22:43:21.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:43:22.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 299 -eq 0 ']' 2020-03-11T22:43:22.485 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:43:22.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 154618822660 -lt 154618822659 2020-03-11T22:43:22.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:43:22.828 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:43:22.828 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:43:22.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:43:22.829 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:43:22.829 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:43:22.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:43:22.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:43:22.830 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:43:22.831 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 2020-03-11T22:43:23.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672975 -lt 2020-03-11T22:43:23.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672975: unary operator expected 2020-03-11T22:43:23.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:43:23.172 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-158913789955 2020-03-11T22:43:23.172 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:43:23.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:43:23.173 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-158913789955 2020-03-11T22:43:23.173 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:43:23.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=158913789955 2020-03-11T22:43:23.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 158913789955' 2020-03-11T22:43:23.175 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:43:23.175 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 158913789955 2020-03-11T22:43:23.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 158913789956 -lt 158913789955 2020-03-11T22:43:23.517 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:43:23.517 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:23.517 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:23.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 32 == 0 2020-03-11T22:43:23.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:23.959 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:23.959 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:23.960 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:23.960 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:23.960 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:23.960 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:24.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:24.296 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:24.296 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:24.296 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:24.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:43:24.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:43:24.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:43:24.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:43:24.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:43:24.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:43:24.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:43:24.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:24.840 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:24.840 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:24.840 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:24.840 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:24.840 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:24.841 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:25.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:25.176 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:25.176 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:25.176 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:25.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:43:25.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:43:25.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:43:25.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:43:25.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:43:25.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:43:25.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:43:25.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:43:25.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:43:26.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:43:26.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:43:26.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:43:26.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:43:26.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:43:26.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:43:26.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:26.294 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:26.294 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:26.294 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:26.295 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:26.295 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:26.295 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:26.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:26.627 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:26.628 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:26.628 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:27.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:43:27.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:43:27.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:43:27.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:43:27.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:43:27.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:43:27.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:43:27.070 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:43:27.070 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:43:27.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:43:27.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:43:27.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:43:27.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:43:27.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:43:27.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:43:27.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:27.915 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:27.915 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:27.916 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:27.916 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:27.916 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:27.916 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:28.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:28.251 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:28.251 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:28.252 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:28.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:43:28.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:43:28.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:43:28.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:43:28.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:43:28.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:43:28.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:43:28.694 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:43:28.695 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:43:29.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:43:29.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:43:29.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:43:29.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:43:29.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:43:29.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:43:29.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:29.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:29.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:29.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:29.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:29.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:29.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:30.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:30.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:30.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:30.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:30.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:43:30.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:43:30.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:43:30.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:43:30.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:43:30.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:43:30.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:43:30.728 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:43:30.728 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:43:31.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:43:31.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:43:31.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:43:31.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:43:31.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:43:32.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:43:32.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:32.774 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:32.774 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:32.775 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:32.775 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:32.775 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:32.775 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:33.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:33.116 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:33.116 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:33.116 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:33.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:43:33.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:43:33.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:43:33.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:43:33.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:43:33.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:43:33.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:43:33.559 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:43:33.559 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:43:34.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:43:34.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:43:34.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:43:34.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:43:34.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:43:37.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:43:37.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:37.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:37.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:37.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:37.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:37.206 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:37.206 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:37.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:37.552 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:37.552 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:37.552 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:38.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:43:38.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:43:38.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:43:38.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:43:38.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:43:38.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:43:38.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:43:38.005 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:43:38.005 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:43:38.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:43:38.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:43:38.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:43:38.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:43:38.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:43:44.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:43:44.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:44.847 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:44.847 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:44.847 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:44.847 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:44.847 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:44.847 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:45.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:45.191 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:45.192 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:45.192 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:45.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:43:45.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:43:45.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:43:45.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:43:45.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:43:45.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:43:45.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:43:45.649 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:43:45.649 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:43:46.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:43:46.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:43:46.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:43:46.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:43:46.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:43:53.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:43:53.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:53.407 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:53.407 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:53.407 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:53.408 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:53.408 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:53.408 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:53.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:53.753 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:53.754 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:53.754 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:54.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:43:54.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:43:54.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:43:54.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:43:54.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:43:54.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:43:54.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:43:54.198 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:43:54.199 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:43:54.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:43:54.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:43:54.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:43:54.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:43:55.070 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:43:55.070 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_fingerprint": "9c96a21d-af8e-49c8-8736-1bc757ebabc1", 2020-03-11T22:43:55.070 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:43:55.071 INFO:tasks.workunit.client.0.smithi145.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:43:55.071 INFO:tasks.workunit.client.0.smithi145.stdout: "timestamp": "2020-03-11T22:43:55.067103+0000", 2020-03-11T22:43:55.071 INFO:tasks.workunit.client.0.smithi145.stdout: "tag": "", 2020-03-11T22:43:55.071 INFO:tasks.workunit.client.0.smithi145.stdout: "health": { 2020-03-11T22:43:55.071 INFO:tasks.workunit.client.0.smithi145.stdout: "status": "HEALTH_WARN", 2020-03-11T22:43:55.071 INFO:tasks.workunit.client.0.smithi145.stdout: "checks": { 2020-03-11T22:43:55.072 INFO:tasks.workunit.client.0.smithi145.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:43:55.072 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:43:55.072 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2020-03-11T22:43:55.072 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:43:55.072 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 17 2020-03-11T22:43:55.072 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.072 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [], 2020-03-11T22:43:55.073 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2020-03-11T22:43:55.073 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.073 INFO:tasks.workunit.client.0.smithi145.stdout: "OSD_DOWN": { 2020-03-11T22:43:55.073 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:43:55.073 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2020-03-11T22:43:55.073 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "1 osds down", 2020-03-11T22:43:55.074 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 1 2020-03-11T22:43:55.074 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.074 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2020-03-11T22:43:55.074 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.074 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "osd.1 (root=default,host=smithi145) is down" 2020-03-11T22:43:55.074 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:43:55.074 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:43:55.075 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2020-03-11T22:43:55.075 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.075 INFO:tasks.workunit.client.0.smithi145.stdout: "PG_DEGRADED": { 2020-03-11T22:43:55.075 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:43:55.075 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2020-03-11T22:43:55.075 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 32 pgs degraded", 2020-03-11T22:43:55.075 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 32 2020-03-11T22:43:55.075 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.076 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2020-03-11T22:43:55.076 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.076 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:43:55.076 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.076 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.076 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.1 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:43:55.076 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.077 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.077 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:43:55.077 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.077 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.077 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:43:55.077 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.077 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.078 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.4 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:43:55.078 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.078 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.078 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.5 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:43:55.079 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.079 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.079 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.6 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:43:55.079 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.079 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.079 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.7 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:43:55.079 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.080 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.080 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.8 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:43:55.080 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.080 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.080 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.9 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:43:55.080 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.081 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.081 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.a is active+undersized+degraded, acting [0,2]" 2020-03-11T22:43:55.081 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.081 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.081 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.b is active+undersized+degraded, acting [0,2]" 2020-03-11T22:43:55.081 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.082 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.082 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.c is active+undersized+degraded, acting [0,2]" 2020-03-11T22:43:55.082 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.082 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.082 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.d is active+undersized+degraded, acting [0,2]" 2020-03-11T22:43:55.082 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.082 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.083 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.e is active+undersized+degraded, acting [0,2]" 2020-03-11T22:43:55.083 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.083 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.083 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.f is active+undersized+degraded, acting [0,2]" 2020-03-11T22:43:55.083 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.083 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.083 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.10 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:43:55.083 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.084 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.084 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.11 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:43:55.084 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.084 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.084 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.12 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:43:55.084 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.084 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.085 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.13 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:43:55.085 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.085 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.085 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.14 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:43:55.085 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.085 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.085 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.15 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:43:55.085 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.086 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.086 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.16 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:43:55.086 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.086 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.086 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.17 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:43:55.086 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.086 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.087 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.18 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:43:55.087 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.087 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.087 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.19 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:43:55.087 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.088 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.088 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.1a is active+undersized+degraded, acting [0,2]" 2020-03-11T22:43:55.088 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.088 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.088 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.1b is active+undersized+degraded, acting [0,2]" 2020-03-11T22:43:55.088 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.088 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.089 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.1c is active+undersized+degraded, acting [0,2]" 2020-03-11T22:43:55.089 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.089 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.089 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.1d is active+undersized+degraded, acting [0,2]" 2020-03-11T22:43:55.089 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.089 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.089 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.1e is active+undersized+degraded, acting [0,2]" 2020-03-11T22:43:55.089 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.090 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.090 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.1f is active+undersized+degraded, acting [0,2]" 2020-03-11T22:43:55.090 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:43:55.090 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:43:55.090 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2020-03-11T22:43:55.090 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.090 INFO:tasks.workunit.client.0.smithi145.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:43:55.091 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:43:55.091 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2020-03-11T22:43:55.091 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:43:55.091 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 1 2020-03-11T22:43:55.091 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.091 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2020-03-11T22:43:55.091 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.091 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:43:55.092 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.092 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.092 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:43:55.092 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:43:55.092 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:43:55.092 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2020-03-11T22:43:55.092 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:43:55.093 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.093 INFO:tasks.workunit.client.0.smithi145.stdout: "mutes": [] 2020-03-11T22:43:55.093 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.093 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap_first_committed": 1, 2020-03-11T22:43:55.093 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap_last_committed": 1, 2020-03-11T22:43:55.093 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap": { 2020-03-11T22:43:55.093 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 1, 2020-03-11T22:43:55.094 INFO:tasks.workunit.client.0.smithi145.stdout: "fsid": "e665ac71-d8d3-412f-b706-8f1d72482f80", 2020-03-11T22:43:55.094 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2020-03-11T22:42:15.092934Z", 2020-03-11T22:43:55.094 INFO:tasks.workunit.client.0.smithi145.stdout: "created": "2020-03-11T22:42:15.092934Z", 2020-03-11T22:43:55.094 INFO:tasks.workunit.client.0.smithi145.stdout: "min_mon_release": 15, 2020-03-11T22:43:55.094 INFO:tasks.workunit.client.0.smithi145.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:43:55.094 INFO:tasks.workunit.client.0.smithi145.stdout: "features": { 2020-03-11T22:43:55.094 INFO:tasks.workunit.client.0.smithi145.stdout: "persistent": [ 2020-03-11T22:43:55.094 INFO:tasks.workunit.client.0.smithi145.stdout: "kraken", 2020-03-11T22:43:55.095 INFO:tasks.workunit.client.0.smithi145.stdout: "luminous", 2020-03-11T22:43:55.095 INFO:tasks.workunit.client.0.smithi145.stdout: "mimic", 2020-03-11T22:43:55.095 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap-prune", 2020-03-11T22:43:55.095 INFO:tasks.workunit.client.0.smithi145.stdout: "nautilus", 2020-03-11T22:43:55.095 INFO:tasks.workunit.client.0.smithi145.stdout: "octopus" 2020-03-11T22:43:55.095 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:43:55.096 INFO:tasks.workunit.client.0.smithi145.stdout: "optional": [] 2020-03-11T22:43:55.096 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.096 INFO:tasks.workunit.client.0.smithi145.stdout: "mons": [ 2020-03-11T22:43:55.096 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.096 INFO:tasks.workunit.client.0.smithi145.stdout: "rank": 0, 2020-03-11T22:43:55.096 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "a", 2020-03-11T22:43:55.096 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2020-03-11T22:43:55.097 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:43:55.097 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.097 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:43:55.097 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:43:55.097 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 0 2020-03-11T22:43:55.097 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:43:55.098 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:43:55.098 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.098 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:43:55.098 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:43:55.098 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 0, 2020-03-11T22:43:55.098 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 0 2020-03-11T22:43:55.098 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:43:55.098 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:43:55.099 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.099 INFO:tasks.workunit.client.0.smithi145.stdout: "quorum": [ 2020-03-11T22:43:55.099 INFO:tasks.workunit.client.0.smithi145.stdout: 0 2020-03-11T22:43:55.099 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:43:55.099 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap": { 2020-03-11T22:43:55.099 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 43, 2020-03-11T22:43:55.099 INFO:tasks.workunit.client.0.smithi145.stdout: "fsid": "e665ac71-d8d3-412f-b706-8f1d72482f80", 2020-03-11T22:43:55.100 INFO:tasks.workunit.client.0.smithi145.stdout: "created": "2020-03-11T22:42:15.727479+0000", 2020-03-11T22:43:55.100 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2020-03-11T22:43:21.637157+0000", 2020-03-11T22:43:55.100 INFO:tasks.workunit.client.0.smithi145.stdout: "last_up_change": "2020-03-11T22:43:17.298677+0000", 2020-03-11T22:43:55.100 INFO:tasks.workunit.client.0.smithi145.stdout: "last_in_change": "2020-03-11T22:43:18.717475+0000", 2020-03-11T22:43:55.100 INFO:tasks.workunit.client.0.smithi145.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:43:55.100 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_num": 5800384, 2020-03-11T22:43:55.100 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_set": [ 2020-03-11T22:43:55.101 INFO:tasks.workunit.client.0.smithi145.stdout: "nodown", 2020-03-11T22:43:55.101 INFO:tasks.workunit.client.0.smithi145.stdout: "noin", 2020-03-11T22:43:55.101 INFO:tasks.workunit.client.0.smithi145.stdout: "noout", 2020-03-11T22:43:55.101 INFO:tasks.workunit.client.0.smithi145.stdout: "pglog_hardlimit", 2020-03-11T22:43:55.101 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snapdirs", 2020-03-11T22:43:55.101 INFO:tasks.workunit.client.0.smithi145.stdout: "recovery_deletes", 2020-03-11T22:43:55.101 INFO:tasks.workunit.client.0.smithi145.stdout: "sortbitwise" 2020-03-11T22:43:55.102 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:43:55.102 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_version": 7, 2020-03-11T22:43:55.102 INFO:tasks.workunit.client.0.smithi145.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:43:55.102 INFO:tasks.workunit.client.0.smithi145.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:43:55.102 INFO:tasks.workunit.client.0.smithi145.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:43:55.102 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_snapshot": "", 2020-03-11T22:43:55.103 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_max": 1, 2020-03-11T22:43:55.103 INFO:tasks.workunit.client.0.smithi145.stdout: "max_osd": 3, 2020-03-11T22:43:55.103 INFO:tasks.workunit.client.0.smithi145.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:43:55.103 INFO:tasks.workunit.client.0.smithi145.stdout: "min_compat_client": "jewel", 2020-03-11T22:43:55.103 INFO:tasks.workunit.client.0.smithi145.stdout: "require_osd_release": "octopus", 2020-03-11T22:43:55.103 INFO:tasks.workunit.client.0.smithi145.stdout: "pools": [ 2020-03-11T22:43:55.103 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.104 INFO:tasks.workunit.client.0.smithi145.stdout: "pool": 1, 2020-03-11T22:43:55.104 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_name": "test", 2020-03-11T22:43:55.104 INFO:tasks.workunit.client.0.smithi145.stdout: "create_time": "2020-03-11T22:42:36.437686+0000", 2020-03-11T22:43:55.104 INFO:tasks.workunit.client.0.smithi145.stdout: "flags": 1, 2020-03-11T22:43:55.104 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_names": "hashpspool", 2020-03-11T22:43:55.104 INFO:tasks.workunit.client.0.smithi145.stdout: "type": 1, 2020-03-11T22:43:55.104 INFO:tasks.workunit.client.0.smithi145.stdout: "size": 3, 2020-03-11T22:43:55.105 INFO:tasks.workunit.client.0.smithi145.stdout: "min_size": 2, 2020-03-11T22:43:55.105 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_rule": 0, 2020-03-11T22:43:55.105 INFO:tasks.workunit.client.0.smithi145.stdout: "object_hash": 2, 2020-03-11T22:43:55.105 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_autoscale_mode": "on", 2020-03-11T22:43:55.105 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num": 32, 2020-03-11T22:43:55.105 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_placement_num": 1, 2020-03-11T22:43:55.105 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_placement_num_target": 32, 2020-03-11T22:43:55.105 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num_target": 32, 2020-03-11T22:43:55.106 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num_pending": 32, 2020-03-11T22:43:55.106 INFO:tasks.workunit.client.0.smithi145.stdout: "last_pg_merge_meta": { 2020-03-11T22:43:55.106 INFO:tasks.workunit.client.0.smithi145.stdout: "source_pgid": "0.0", 2020-03-11T22:43:55.106 INFO:tasks.workunit.client.0.smithi145.stdout: "ready_epoch": 0, 2020-03-11T22:43:55.106 INFO:tasks.workunit.client.0.smithi145.stdout: "last_epoch_started": 0, 2020-03-11T22:43:55.106 INFO:tasks.workunit.client.0.smithi145.stdout: "last_epoch_clean": 0, 2020-03-11T22:43:55.106 INFO:tasks.workunit.client.0.smithi145.stdout: "source_version": "0'0", 2020-03-11T22:43:55.107 INFO:tasks.workunit.client.0.smithi145.stdout: "target_version": "0'0" 2020-03-11T22:43:55.107 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.107 INFO:tasks.workunit.client.0.smithi145.stdout: "last_change": "42", 2020-03-11T22:43:55.107 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend": "0", 2020-03-11T22:43:55.107 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:43:55.107 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend_preluminous": "42", 2020-03-11T22:43:55.107 INFO:tasks.workunit.client.0.smithi145.stdout: "auid": 0, 2020-03-11T22:43:55.108 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_mode": "selfmanaged", 2020-03-11T22:43:55.108 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_seq": 0, 2020-03-11T22:43:55.108 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_epoch": 0, 2020-03-11T22:43:55.108 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_snaps": [], 2020-03-11T22:43:55.108 INFO:tasks.workunit.client.0.smithi145.stdout: "removed_snaps": "[]", 2020-03-11T22:43:55.108 INFO:tasks.workunit.client.0.smithi145.stdout: "quota_max_bytes": 0, 2020-03-11T22:43:55.108 INFO:tasks.workunit.client.0.smithi145.stdout: "quota_max_objects": 0, 2020-03-11T22:43:55.108 INFO:tasks.workunit.client.0.smithi145.stdout: "tiers": [], 2020-03-11T22:43:55.109 INFO:tasks.workunit.client.0.smithi145.stdout: "tier_of": -1, 2020-03-11T22:43:55.109 INFO:tasks.workunit.client.0.smithi145.stdout: "read_tier": -1, 2020-03-11T22:43:55.109 INFO:tasks.workunit.client.0.smithi145.stdout: "write_tier": -1, 2020-03-11T22:43:55.109 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_mode": "none", 2020-03-11T22:43:55.109 INFO:tasks.workunit.client.0.smithi145.stdout: "target_max_bytes": 0, 2020-03-11T22:43:55.109 INFO:tasks.workunit.client.0.smithi145.stdout: "target_max_objects": 0, 2020-03-11T22:43:55.109 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:43:55.110 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:43:55.110 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:43:55.110 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_min_flush_age": 0, 2020-03-11T22:43:55.110 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_min_evict_age": 0, 2020-03-11T22:43:55.110 INFO:tasks.workunit.client.0.smithi145.stdout: "erasure_code_profile": "", 2020-03-11T22:43:55.110 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_params": { 2020-03-11T22:43:55.110 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "none" 2020-03-11T22:43:55.111 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.111 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_period": 0, 2020-03-11T22:43:55.111 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_count": 0, 2020-03-11T22:43:55.111 INFO:tasks.workunit.client.0.smithi145.stdout: "use_gmt_hitset": true, 2020-03-11T22:43:55.111 INFO:tasks.workunit.client.0.smithi145.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:43:55.111 INFO:tasks.workunit.client.0.smithi145.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:43:55.111 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:43:55.112 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:43:55.112 INFO:tasks.workunit.client.0.smithi145.stdout: "grade_table": [], 2020-03-11T22:43:55.112 INFO:tasks.workunit.client.0.smithi145.stdout: "stripe_width": 0, 2020-03-11T22:43:55.112 INFO:tasks.workunit.client.0.smithi145.stdout: "expected_num_objects": 0, 2020-03-11T22:43:55.112 INFO:tasks.workunit.client.0.smithi145.stdout: "fast_read": false, 2020-03-11T22:43:55.113 INFO:tasks.workunit.client.0.smithi145.stdout: "options": {}, 2020-03-11T22:43:55.113 INFO:tasks.workunit.client.0.smithi145.stdout: "application_metadata": {} 2020-03-11T22:43:55.113 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:43:55.113 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:43:55.113 INFO:tasks.workunit.client.0.smithi145.stdout: "osds": [ 2020-03-11T22:43:55.114 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.114 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2020-03-11T22:43:55.114 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "25919b39-e67c-442a-a46f-e5d1e5ebdbd6", 2020-03-11T22:43:55.114 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 1, 2020-03-11T22:43:55.115 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2020-03-11T22:43:55.115 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2020-03-11T22:43:55.115 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2020-03-11T22:43:55.115 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 5, 2020-03-11T22:43:55.115 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 30, 2020-03-11T22:43:55.115 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 36, 2020-03-11T22:43:55.115 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 42, 2020-03-11T22:43:55.116 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 31, 2020-03-11T22:43:55.116 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2020-03-11T22:43:55.116 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2020-03-11T22:43:55.116 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:43:55.116 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.116 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:43:55.116 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:43:55.117 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 39271 2020-03-11T22:43:55.117 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.117 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.117 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:43:55.117 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:43:55.117 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 39271 2020-03-11T22:43:55.117 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:43:55.118 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:43:55.118 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.118 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2020-03-11T22:43:55.118 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:43:55.118 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.118 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:43:55.118 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:43:55.118 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 39271 2020-03-11T22:43:55.119 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.119 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.119 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:43:55.119 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:43:55.119 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 39271 2020-03-11T22:43:55.119 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:43:55.120 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:43:55.120 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.120 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2020-03-11T22:43:55.120 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:43:55.120 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.120 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:43:55.120 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:43:55.121 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 39271 2020-03-11T22:43:55.121 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.121 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.121 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:43:55.121 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:43:55.121 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 39271 2020-03-11T22:43:55.121 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:43:55.122 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:43:55.122 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.122 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2020-03-11T22:43:55.122 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:43:55.122 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.122 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:43:55.122 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:43:55.123 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 39271 2020-03-11T22:43:55.123 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.123 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.123 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:43:55.123 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:43:55.123 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 39271 2020-03-11T22:43:55.124 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:43:55.124 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:43:55.124 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.124 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6803/39271", 2020-03-11T22:43:55.124 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6805/39271", 2020-03-11T22:43:55.124 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6809/39271", 2020-03-11T22:43:55.124 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6807/39271", 2020-03-11T22:43:55.125 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2020-03-11T22:43:55.125 INFO:tasks.workunit.client.0.smithi145.stdout: "exists", 2020-03-11T22:43:55.125 INFO:tasks.workunit.client.0.smithi145.stdout: "up" 2020-03-11T22:43:55.125 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:43:55.125 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.126 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.126 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2020-03-11T22:43:55.126 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "72907ff7-85ea-4a4a-b62c-397e4c805acb", 2020-03-11T22:43:55.126 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 0, 2020-03-11T22:43:55.127 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2020-03-11T22:43:55.127 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2020-03-11T22:43:55.127 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2020-03-11T22:43:55.127 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 0, 2020-03-11T22:43:55.127 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 0, 2020-03-11T22:43:55.127 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 10, 2020-03-11T22:43:55.128 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 22, 2020-03-11T22:43:55.128 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 32, 2020-03-11T22:43:55.128 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2020-03-11T22:43:55.128 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2020-03-11T22:43:55.128 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:43:55.128 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.128 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:43:55.129 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:43:55.129 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 34828 2020-03-11T22:43:55.129 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.129 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.129 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:43:55.129 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:43:55.129 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 34828 2020-03-11T22:43:55.130 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:43:55.130 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:43:55.130 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.130 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2020-03-11T22:43:55.130 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:43:55.130 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.130 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:43:55.131 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:43:55.131 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 34828 2020-03-11T22:43:55.131 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.131 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.131 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:43:55.131 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:43:55.131 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 34828 2020-03-11T22:43:55.131 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:43:55.132 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:43:55.132 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.132 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2020-03-11T22:43:55.132 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:43:55.132 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.132 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:43:55.132 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:43:55.133 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 34828 2020-03-11T22:43:55.133 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.133 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.133 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:43:55.133 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:43:55.133 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 34828 2020-03-11T22:43:55.133 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:43:55.134 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:43:55.134 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.134 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2020-03-11T22:43:55.134 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:43:55.134 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.134 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:43:55.134 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:43:55.134 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 34828 2020-03-11T22:43:55.135 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.135 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.135 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:43:55.135 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:43:55.135 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 34828 2020-03-11T22:43:55.135 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:43:55.135 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:43:55.136 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.136 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6811/34828", 2020-03-11T22:43:55.136 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6813/34828", 2020-03-11T22:43:55.136 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6817/34828", 2020-03-11T22:43:55.136 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6815/34828", 2020-03-11T22:43:55.136 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2020-03-11T22:43:55.136 INFO:tasks.workunit.client.0.smithi145.stdout: "exists" 2020-03-11T22:43:55.136 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:43:55.137 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.137 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.137 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2020-03-11T22:43:55.137 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "f1f86676-d3cf-4822-8bec-ab32748439d8", 2020-03-11T22:43:55.137 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 1, 2020-03-11T22:43:55.137 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2020-03-11T22:43:55.137 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2020-03-11T22:43:55.138 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2020-03-11T22:43:55.138 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 15, 2020-03-11T22:43:55.138 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 30, 2020-03-11T22:43:55.138 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 37, 2020-03-11T22:43:55.138 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 0, 2020-03-11T22:43:55.138 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 33, 2020-03-11T22:43:55.138 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2020-03-11T22:43:55.139 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2020-03-11T22:43:55.139 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:43:55.139 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.139 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:43:55.139 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:43:55.139 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 39465 2020-03-11T22:43:55.140 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.140 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.140 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:43:55.140 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:43:55.140 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 39465 2020-03-11T22:43:55.140 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:43:55.140 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:43:55.140 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.141 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2020-03-11T22:43:55.141 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:43:55.141 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.141 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:43:55.141 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:43:55.141 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 39465 2020-03-11T22:43:55.141 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.142 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.142 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:43:55.142 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:43:55.142 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 39465 2020-03-11T22:43:55.142 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:43:55.142 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:43:55.142 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.143 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2020-03-11T22:43:55.143 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:43:55.143 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.143 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:43:55.143 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:43:55.143 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 39465 2020-03-11T22:43:55.143 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.143 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.144 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:43:55.144 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:43:55.144 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 39465 2020-03-11T22:43:55.144 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:43:55.144 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:43:55.144 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.144 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2020-03-11T22:43:55.145 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:43:55.145 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.145 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:43:55.145 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:43:55.145 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 39465 2020-03-11T22:43:55.145 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.146 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.146 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:43:55.146 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:43:55.146 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 39465 2020-03-11T22:43:55.146 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:43:55.146 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:43:55.147 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.147 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6811/39465", 2020-03-11T22:43:55.147 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6813/39465", 2020-03-11T22:43:55.147 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6817/39465", 2020-03-11T22:43:55.147 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6815/39465", 2020-03-11T22:43:55.147 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2020-03-11T22:43:55.148 INFO:tasks.workunit.client.0.smithi145.stdout: "exists", 2020-03-11T22:43:55.148 INFO:tasks.workunit.client.0.smithi145.stdout: "up" 2020-03-11T22:43:55.148 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:43:55.148 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:43:55.148 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:43:55.148 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_xinfo": [ 2020-03-11T22:43:55.148 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.149 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2020-03-11T22:43:55.149 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "2020-03-11T22:43:11.224431+0000", 2020-03-11T22:43:55.149 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2020-03-11T22:43:55.149 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2020-03-11T22:43:55.149 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138292836696063, 2020-03-11T22:43:55.149 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2020-03-11T22:43:55.149 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2020-03-11T22:42:22.238601+0000", 2020-03-11T22:43:55.150 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2020-03-11T22:43:55.150 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.150 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.150 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2020-03-11T22:43:55.150 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "2020-03-11T22:43:11.742976+0000", 2020-03-11T22:43:55.150 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2020-03-11T22:43:55.150 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2020-03-11T22:43:55.151 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138292836696063, 2020-03-11T22:43:55.151 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2020-03-11T22:43:55.151 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2020-03-11T22:42:28.436781+0000", 2020-03-11T22:43:55.151 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2020-03-11T22:43:55.151 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.151 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.151 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2020-03-11T22:43:55.152 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "2020-03-11T22:43:12.241054+0000", 2020-03-11T22:43:55.152 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2020-03-11T22:43:55.152 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2020-03-11T22:43:55.152 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138292836696063, 2020-03-11T22:43:55.152 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2020-03-11T22:43:55.152 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2020-03-11T22:42:34.528134+0000", 2020-03-11T22:43:55.152 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2020-03-11T22:43:55.153 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:43:55.153 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:43:55.153 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_upmap": [], 2020-03-11T22:43:55.153 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_upmap_items": [], 2020-03-11T22:43:55.153 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_temp": [], 2020-03-11T22:43:55.153 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_temp": [], 2020-03-11T22:43:55.153 INFO:tasks.workunit.client.0.smithi145.stdout: "blacklist": {}, 2020-03-11T22:43:55.154 INFO:tasks.workunit.client.0.smithi145.stdout: "erasure_code_profiles": { 2020-03-11T22:43:55.154 INFO:tasks.workunit.client.0.smithi145.stdout: "default": { 2020-03-11T22:43:55.154 INFO:tasks.workunit.client.0.smithi145.stdout: "k": "2", 2020-03-11T22:43:55.154 INFO:tasks.workunit.client.0.smithi145.stdout: "m": "2", 2020-03-11T22:43:55.154 INFO:tasks.workunit.client.0.smithi145.stdout: "plugin": "jerasure", 2020-03-11T22:43:55.154 INFO:tasks.workunit.client.0.smithi145.stdout: "technique": "reed_sol_van" 2020-03-11T22:43:55.154 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:43:55.155 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.155 INFO:tasks.workunit.client.0.smithi145.stdout: "removed_snaps_queue": [], 2020-03-11T22:43:55.155 INFO:tasks.workunit.client.0.smithi145.stdout: "new_removed_snaps": [], 2020-03-11T22:43:55.155 INFO:tasks.workunit.client.0.smithi145.stdout: "new_purged_snaps": [], 2020-03-11T22:43:55.155 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_node_flags": {}, 2020-03-11T22:43:55.155 INFO:tasks.workunit.client.0.smithi145.stdout: "device_class_flags": {} 2020-03-11T22:43:55.155 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.155 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_metadata": [ 2020-03-11T22:43:55.156 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.156 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2020-03-11T22:43:55.156 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2020-03-11T22:43:55.156 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6804/39271,v1:127.0.0.1:6805/39271]", 2020-03-11T22:43:55.156 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2020-03-11T22:43:55.156 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2020-03-11T22:43:55.156 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:43:55.157 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:43:55.157 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:43:55.157 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:43:55.157 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:43:55.157 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:43:55.157 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:43:55.157 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:43:55.158 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:43:55.158 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:43:55.158 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:43:55.158 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:43:55.158 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "octopus", 2020-03-11T22:43:55.158 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:43:55.158 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:43:55.159 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:43:55.159 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2020-03-11T22:43:55.159 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XLDP", 2020-03-11T22:43:55.159 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:43:55.159 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2020-03-11T22:43:55.159 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2020-03-11T22:43:55.159 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:43:55.160 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2020-03-11T22:43:55.160 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6802/39271,v1:127.0.0.1:6803/39271]", 2020-03-11T22:43:55.160 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2020-03-11T22:43:55.160 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/39271,v1:127.0.0.1:6809/39271]", 2020-03-11T22:43:55.160 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/39271,v1:127.0.0.1:6807/39271]", 2020-03-11T22:43:55.160 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2020-03-11T22:43:55.160 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2020-03-11T22:43:55.160 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:43:55.161 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:43:55.161 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2020-03-11T22:43:55.161 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32655820", 2020-03-11T22:43:55.161 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:43:55.161 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:43:55.161 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2020-03-11T22:43:55.161 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:43:55.162 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:43:55.162 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2020-03-11T22:43:55.162 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.162 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.162 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2020-03-11T22:43:55.162 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2020-03-11T22:43:55.162 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6812/34828,v1:127.0.0.1:6813/34828]", 2020-03-11T22:43:55.162 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2020-03-11T22:43:55.163 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2020-03-11T22:43:55.163 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:43:55.163 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:43:55.163 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:43:55.163 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:43:55.163 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:43:55.163 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:43:55.164 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:43:55.164 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:43:55.164 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:43:55.164 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:43:55.164 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:43:55.164 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:43:55.164 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "octopus", 2020-03-11T22:43:55.165 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:43:55.165 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:43:55.165 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:43:55.165 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2020-03-11T22:43:55.165 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XLDP", 2020-03-11T22:43:55.165 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:43:55.165 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2020-03-11T22:43:55.166 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2020-03-11T22:43:55.166 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:43:55.166 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2020-03-11T22:43:55.166 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6810/34828,v1:127.0.0.1:6811/34828]", 2020-03-11T22:43:55.166 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2020-03-11T22:43:55.166 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/34828,v1:127.0.0.1:6817/34828]", 2020-03-11T22:43:55.166 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/34828,v1:127.0.0.1:6815/34828]", 2020-03-11T22:43:55.166 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2020-03-11T22:43:55.167 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2020-03-11T22:43:55.167 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:43:55.167 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:43:55.167 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2020-03-11T22:43:55.167 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32655820", 2020-03-11T22:43:55.167 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:43:55.168 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:43:55.168 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2020-03-11T22:43:55.168 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:43:55.168 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:43:55.168 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2020-03-11T22:43:55.169 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.169 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.169 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2020-03-11T22:43:55.169 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2020-03-11T22:43:55.170 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6812/39465,v1:127.0.0.1:6813/39465]", 2020-03-11T22:43:55.170 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2020-03-11T22:43:55.170 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2020-03-11T22:43:55.170 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:43:55.170 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:43:55.170 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:43:55.170 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:43:55.171 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:43:55.171 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:43:55.171 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:43:55.171 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:43:55.171 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:43:55.171 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:43:55.171 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:43:55.172 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:43:55.172 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "octopus", 2020-03-11T22:43:55.172 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:43:55.172 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:43:55.172 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:43:55.173 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2020-03-11T22:43:55.173 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XLDP", 2020-03-11T22:43:55.173 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:43:55.173 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2020-03-11T22:43:55.173 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2020-03-11T22:43:55.174 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:43:55.174 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2020-03-11T22:43:55.174 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6810/39465,v1:127.0.0.1:6811/39465]", 2020-03-11T22:43:55.174 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2020-03-11T22:43:55.175 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/39465,v1:127.0.0.1:6817/39465]", 2020-03-11T22:43:55.175 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/39465,v1:127.0.0.1:6815/39465]", 2020-03-11T22:43:55.175 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2020-03-11T22:43:55.175 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2020-03-11T22:43:55.176 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:43:55.176 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:43:55.176 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2020-03-11T22:43:55.176 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32655820", 2020-03-11T22:43:55.177 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:43:55.177 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:43:55.177 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2020-03-11T22:43:55.177 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:43:55.177 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:43:55.178 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2020-03-11T22:43:55.178 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:43:55.178 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:43:55.178 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap_first_committed": 1, 2020-03-11T22:43:55.179 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap_last_committed": 43, 2020-03-11T22:43:55.179 INFO:tasks.workunit.client.0.smithi145.stdout: "crushmap": { 2020-03-11T22:43:55.179 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": [ 2020-03-11T22:43:55.179 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.179 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2020-03-11T22:43:55.180 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.0", 2020-03-11T22:43:55.180 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2020-03-11T22:43:55.180 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.180 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.180 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2020-03-11T22:43:55.180 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.1", 2020-03-11T22:43:55.181 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2020-03-11T22:43:55.181 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.181 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.181 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2020-03-11T22:43:55.181 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.2", 2020-03-11T22:43:55.182 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2020-03-11T22:43:55.182 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:43:55.182 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:43:55.182 INFO:tasks.workunit.client.0.smithi145.stdout: "types": [ 2020-03-11T22:43:55.182 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.182 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 0, 2020-03-11T22:43:55.182 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd" 2020-03-11T22:43:55.183 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.183 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.183 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2020-03-11T22:43:55.183 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "host" 2020-03-11T22:43:55.183 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.183 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.183 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 2, 2020-03-11T22:43:55.184 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "chassis" 2020-03-11T22:43:55.184 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.184 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.184 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 3, 2020-03-11T22:43:55.184 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "rack" 2020-03-11T22:43:55.184 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.184 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.184 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 4, 2020-03-11T22:43:55.185 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "row" 2020-03-11T22:43:55.185 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.185 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.185 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 5, 2020-03-11T22:43:55.185 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "pdu" 2020-03-11T22:43:55.185 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.185 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.186 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 6, 2020-03-11T22:43:55.186 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "pod" 2020-03-11T22:43:55.186 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.186 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.186 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 7, 2020-03-11T22:43:55.187 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "room" 2020-03-11T22:43:55.187 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.187 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.187 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 8, 2020-03-11T22:43:55.187 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "datacenter" 2020-03-11T22:43:55.187 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.187 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.187 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 9, 2020-03-11T22:43:55.188 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "zone" 2020-03-11T22:43:55.188 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.188 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.188 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 10, 2020-03-11T22:43:55.188 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "region" 2020-03-11T22:43:55.188 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.188 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.189 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2020-03-11T22:43:55.189 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "root" 2020-03-11T22:43:55.189 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:43:55.189 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:43:55.189 INFO:tasks.workunit.client.0.smithi145.stdout: "buckets": [ 2020-03-11T22:43:55.189 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.189 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -1, 2020-03-11T22:43:55.190 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "default", 2020-03-11T22:43:55.190 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2020-03-11T22:43:55.190 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "root", 2020-03-11T22:43:55.190 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2020-03-11T22:43:55.190 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2020-03-11T22:43:55.190 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2020-03-11T22:43:55.190 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:43:55.191 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.191 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -3, 2020-03-11T22:43:55.191 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2020-03-11T22:43:55.191 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2020-03-11T22:43:55.191 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:43:55.191 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:43:55.191 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.192 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.192 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -2, 2020-03-11T22:43:55.192 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "default~hdd", 2020-03-11T22:43:55.192 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2020-03-11T22:43:55.192 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "root", 2020-03-11T22:43:55.192 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2020-03-11T22:43:55.192 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2020-03-11T22:43:55.192 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2020-03-11T22:43:55.193 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:43:55.193 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.194 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -4, 2020-03-11T22:43:55.194 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2020-03-11T22:43:55.194 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2020-03-11T22:43:55.194 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:43:55.194 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:43:55.194 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.195 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.195 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -3, 2020-03-11T22:43:55.195 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "smithi145", 2020-03-11T22:43:55.195 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2020-03-11T22:43:55.195 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "host", 2020-03-11T22:43:55.195 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2020-03-11T22:43:55.195 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2020-03-11T22:43:55.195 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2020-03-11T22:43:55.196 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:43:55.196 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.196 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2020-03-11T22:43:55.196 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2020-03-11T22:43:55.196 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2020-03-11T22:43:55.196 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.196 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.197 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2020-03-11T22:43:55.197 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2020-03-11T22:43:55.197 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 1 2020-03-11T22:43:55.197 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.197 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.197 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2020-03-11T22:43:55.197 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2020-03-11T22:43:55.198 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 2 2020-03-11T22:43:55.198 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:43:55.198 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:43:55.198 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.198 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.198 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -4, 2020-03-11T22:43:55.198 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "smithi145~hdd", 2020-03-11T22:43:55.198 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2020-03-11T22:43:55.199 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "host", 2020-03-11T22:43:55.199 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2020-03-11T22:43:55.199 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2020-03-11T22:43:55.199 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2020-03-11T22:43:55.199 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:43:55.199 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.199 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2020-03-11T22:43:55.199 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2020-03-11T22:43:55.200 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2020-03-11T22:43:55.200 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.200 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.200 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2020-03-11T22:43:55.200 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2020-03-11T22:43:55.200 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 1 2020-03-11T22:43:55.200 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.201 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.201 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2020-03-11T22:43:55.201 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2020-03-11T22:43:55.201 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 2 2020-03-11T22:43:55.201 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:43:55.201 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:43:55.202 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:43:55.202 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:43:55.202 INFO:tasks.workunit.client.0.smithi145.stdout: "rules": [ 2020-03-11T22:43:55.202 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.202 INFO:tasks.workunit.client.0.smithi145.stdout: "rule_id": 0, 2020-03-11T22:43:55.202 INFO:tasks.workunit.client.0.smithi145.stdout: "rule_name": "replicated_rule", 2020-03-11T22:43:55.202 INFO:tasks.workunit.client.0.smithi145.stdout: "ruleset": 0, 2020-03-11T22:43:55.203 INFO:tasks.workunit.client.0.smithi145.stdout: "type": 1, 2020-03-11T22:43:55.203 INFO:tasks.workunit.client.0.smithi145.stdout: "min_size": 1, 2020-03-11T22:43:55.203 INFO:tasks.workunit.client.0.smithi145.stdout: "max_size": 10, 2020-03-11T22:43:55.203 INFO:tasks.workunit.client.0.smithi145.stdout: "steps": [ 2020-03-11T22:43:55.203 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.203 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "take", 2020-03-11T22:43:55.203 INFO:tasks.workunit.client.0.smithi145.stdout: "item": -1, 2020-03-11T22:43:55.204 INFO:tasks.workunit.client.0.smithi145.stdout: "item_name": "default" 2020-03-11T22:43:55.204 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.204 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.204 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "choose_firstn", 2020-03-11T22:43:55.204 INFO:tasks.workunit.client.0.smithi145.stdout: "num": 0, 2020-03-11T22:43:55.204 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "osd" 2020-03-11T22:43:55.204 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.204 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.205 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "emit" 2020-03-11T22:43:55.205 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:43:55.205 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:43:55.205 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:43:55.206 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:43:55.206 INFO:tasks.workunit.client.0.smithi145.stdout: "tunables": { 2020-03-11T22:43:55.206 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_local_tries": 0, 2020-03-11T22:43:55.206 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:43:55.206 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_total_tries": 50, 2020-03-11T22:43:55.207 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:43:55.207 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:43:55.207 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_stable": 1, 2020-03-11T22:43:55.207 INFO:tasks.workunit.client.0.smithi145.stdout: "straw_calc_version": 1, 2020-03-11T22:43:55.207 INFO:tasks.workunit.client.0.smithi145.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:43:55.207 INFO:tasks.workunit.client.0.smithi145.stdout: "profile": "jewel", 2020-03-11T22:43:55.207 INFO:tasks.workunit.client.0.smithi145.stdout: "optimal_tunables": 1, 2020-03-11T22:43:55.207 INFO:tasks.workunit.client.0.smithi145.stdout: "legacy_tunables": 0, 2020-03-11T22:43:55.208 INFO:tasks.workunit.client.0.smithi145.stdout: "minimum_required_version": "jewel", 2020-03-11T22:43:55.208 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables": 1, 2020-03-11T22:43:55.208 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables2": 1, 2020-03-11T22:43:55.208 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v2_rules": 0, 2020-03-11T22:43:55.208 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables3": 1, 2020-03-11T22:43:55.208 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v3_rules": 0, 2020-03-11T22:43:55.208 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v4_buckets": 1, 2020-03-11T22:43:55.209 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables5": 1, 2020-03-11T22:43:55.209 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v5_rules": 0 2020-03-11T22:43:55.209 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.209 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_args": {} 2020-03-11T22:43:55.209 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.209 INFO:tasks.workunit.client.0.smithi145.stdout: "fsmap": { 2020-03-11T22:43:55.209 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 1, 2020-03-11T22:43:55.210 INFO:tasks.workunit.client.0.smithi145.stdout: "default_fscid": -1, 2020-03-11T22:43:55.210 INFO:tasks.workunit.client.0.smithi145.stdout: "compat": { 2020-03-11T22:43:55.210 INFO:tasks.workunit.client.0.smithi145.stdout: "compat": {}, 2020-03-11T22:43:55.210 INFO:tasks.workunit.client.0.smithi145.stdout: "ro_compat": {}, 2020-03-11T22:43:55.210 INFO:tasks.workunit.client.0.smithi145.stdout: "incompat": { 2020-03-11T22:43:55.210 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_1": "base v0.20", 2020-03-11T22:43:55.210 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:43:55.211 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:43:55.211 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:43:55.211 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:43:55.211 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:43:55.211 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_8": "no anchor table", 2020-03-11T22:43:55.211 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_9": "file layout v2", 2020-03-11T22:43:55.211 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:43:55.212 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:43:55.212 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.212 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_flags": { 2020-03-11T22:43:55.212 INFO:tasks.workunit.client.0.smithi145.stdout: "enable_multiple": false, 2020-03-11T22:43:55.212 INFO:tasks.workunit.client.0.smithi145.stdout: "ever_enabled_multiple": false 2020-03-11T22:43:55.212 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.212 INFO:tasks.workunit.client.0.smithi145.stdout: "standbys": [], 2020-03-11T22:43:55.213 INFO:tasks.workunit.client.0.smithi145.stdout: "filesystems": [] 2020-03-11T22:43:55.213 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.213 INFO:tasks.workunit.client.0.smithi145.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:43:55.213 INFO:tasks.workunit.client.0.smithi145.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:43:55.213 INFO:tasks.workunit.client.0.smithi145.stdout: "auth": { 2020-03-11T22:43:55.213 INFO:tasks.workunit.client.0.smithi145.stdout: "first_committed": 1, 2020-03-11T22:43:55.213 INFO:tasks.workunit.client.0.smithi145.stdout: "last_committed": 5, 2020-03-11T22:43:55.214 INFO:tasks.workunit.client.0.smithi145.stdout: "num_secrets": 3 2020-03-11T22:43:55.214 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.214 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg": 32, 2020-03-11T22:43:55.214 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_active": 32, 2020-03-11T22:43:55.214 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_unknown": 0, 2020-03-11T22:43:55.214 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osd": 3, 2020-03-11T22:43:55.214 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_sum": { 2020-03-11T22:43:55.215 INFO:tasks.workunit.client.0.smithi145.stdout: "stat_sum": { 2020-03-11T22:43:55.215 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes": 48300, 2020-03-11T22:43:55.215 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects": 100, 2020-03-11T22:43:55.215 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_clones": 0, 2020-03-11T22:43:55.215 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_copies": 300, 2020-03-11T22:43:55.215 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:43:55.216 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing": 0, 2020-03-11T22:43:55.216 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_degraded": 100, 2020-03-11T22:43:55.216 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_misplaced": 0, 2020-03-11T22:43:55.216 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_unfound": 0, 2020-03-11T22:43:55.216 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_dirty": 100, 2020-03-11T22:43:55.216 INFO:tasks.workunit.client.0.smithi145.stdout: "num_whiteouts": 0, 2020-03-11T22:43:55.216 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read": 0, 2020-03-11T22:43:55.217 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read_kb": 0, 2020-03-11T22:43:55.217 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write": 100, 2020-03-11T22:43:55.217 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write_kb": 100, 2020-03-11T22:43:55.217 INFO:tasks.workunit.client.0.smithi145.stdout: "num_scrub_errors": 0, 2020-03-11T22:43:55.217 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:43:55.217 INFO:tasks.workunit.client.0.smithi145.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:43:55.217 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_recovered": 0, 2020-03-11T22:43:55.218 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_recovered": 0, 2020-03-11T22:43:55.218 INFO:tasks.workunit.client.0.smithi145.stdout: "num_keys_recovered": 0, 2020-03-11T22:43:55.218 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_omap": 0, 2020-03-11T22:43:55.218 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:43:55.218 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:43:55.218 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush": 0, 2020-03-11T22:43:55.219 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_kb": 0, 2020-03-11T22:43:55.219 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict": 0, 2020-03-11T22:43:55.219 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_kb": 0, 2020-03-11T22:43:55.219 INFO:tasks.workunit.client.0.smithi145.stdout: "num_promote": 0, 2020-03-11T22:43:55.219 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_high": 0, 2020-03-11T22:43:55.219 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_low": 0, 2020-03-11T22:43:55.219 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_some": 0, 2020-03-11T22:43:55.220 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_full": 0, 2020-03-11T22:43:55.220 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_pinned": 0, 2020-03-11T22:43:55.220 INFO:tasks.workunit.client.0.smithi145.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:43:55.220 INFO:tasks.workunit.client.0.smithi145.stdout: "num_large_omap_objects": 0, 2020-03-11T22:43:55.220 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_manifest": 0, 2020-03-11T22:43:55.220 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_bytes": 0, 2020-03-11T22:43:55.221 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_keys": 0, 2020-03-11T22:43:55.221 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_repaired": 0 2020-03-11T22:43:55.221 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.221 INFO:tasks.workunit.client.0.smithi145.stdout: "store_stats": { 2020-03-11T22:43:55.221 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 0, 2020-03-11T22:43:55.221 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 0, 2020-03-11T22:43:55.221 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2020-03-11T22:43:55.221 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 0, 2020-03-11T22:43:55.222 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 0, 2020-03-11T22:43:55.222 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2020-03-11T22:43:55.222 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2020-03-11T22:43:55.222 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2020-03-11T22:43:55.222 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 0, 2020-03-11T22:43:55.222 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 0 2020-03-11T22:43:55.222 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.223 INFO:tasks.workunit.client.0.smithi145.stdout: "log_size": 512, 2020-03-11T22:43:55.223 INFO:tasks.workunit.client.0.smithi145.stdout: "ondisk_log_size": 512, 2020-03-11T22:43:55.223 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 64, 2020-03-11T22:43:55.223 INFO:tasks.workunit.client.0.smithi145.stdout: "acting": 64, 2020-03-11T22:43:55.223 INFO:tasks.workunit.client.0.smithi145.stdout: "num_store_stats": 0 2020-03-11T22:43:55.223 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.224 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_sum": { 2020-03-11T22:43:55.224 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 0, 2020-03-11T22:43:55.224 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 0, 2020-03-11T22:43:55.224 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pgs": 65, 2020-03-11T22:43:55.224 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osds": 3, 2020-03-11T22:43:55.224 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_osds": 3, 2020-03-11T22:43:55.224 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:43:55.224 INFO:tasks.workunit.client.0.smithi145.stdout: "kb": 314572800, 2020-03-11T22:43:55.225 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used": 3180352, 2020-03-11T22:43:55.225 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_data": 34304, 2020-03-11T22:43:55.225 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_omap": 6, 2020-03-11T22:43:55.225 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_meta": 3145721, 2020-03-11T22:43:55.225 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_avail": 311392448, 2020-03-11T22:43:55.225 INFO:tasks.workunit.client.0.smithi145.stdout: "statfs": { 2020-03-11T22:43:55.225 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 322122547200, 2020-03-11T22:43:55.226 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 318865866752, 2020-03-11T22:43:55.226 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 3221225472, 2020-03-11T22:43:55.226 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 35127296, 2020-03-11T22:43:55.226 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 432213, 2020-03-11T22:43:55.226 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2020-03-11T22:43:55.226 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2020-03-11T22:43:55.226 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2020-03-11T22:43:55.226 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 6856, 2020-03-11T22:43:55.227 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 3221218616 2020-03-11T22:43:55.227 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.227 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_peers": [], 2020-03-11T22:43:55.227 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:43:55.227 INFO:tasks.workunit.client.0.smithi145.stdout: "num_snap_trimming": 0, 2020-03-11T22:43:55.227 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shards_repaired": 0, 2020-03-11T22:43:55.227 INFO:tasks.workunit.client.0.smithi145.stdout: "op_queue_age_hist": { 2020-03-11T22:43:55.228 INFO:tasks.workunit.client.0.smithi145.stdout: "histogram": [], 2020-03-11T22:43:55.228 INFO:tasks.workunit.client.0.smithi145.stdout: "upper_bound": 1 2020-03-11T22:43:55.228 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.228 INFO:tasks.workunit.client.0.smithi145.stdout: "perf_stat": { 2020-03-11T22:43:55.228 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ms": 0, 2020-03-11T22:43:55.228 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ms": 0, 2020-03-11T22:43:55.228 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ns": 0, 2020-03-11T22:43:55.228 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ns": 0 2020-03-11T22:43:55.229 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.229 INFO:tasks.workunit.client.0.smithi145.stdout: "alerts": [], 2020-03-11T22:43:55.229 INFO:tasks.workunit.client.0.smithi145.stdout: "network_ping_times": [] 2020-03-11T22:43:55.229 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.229 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_sum_by_class": { 2020-03-11T22:43:55.229 INFO:tasks.workunit.client.0.smithi145.stdout: "hdd": { 2020-03-11T22:43:55.229 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 0, 2020-03-11T22:43:55.230 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 0, 2020-03-11T22:43:55.230 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pgs": 65, 2020-03-11T22:43:55.230 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osds": 3, 2020-03-11T22:43:55.230 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_osds": 3, 2020-03-11T22:43:55.230 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:43:55.230 INFO:tasks.workunit.client.0.smithi145.stdout: "kb": 314572800, 2020-03-11T22:43:55.230 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used": 3180352, 2020-03-11T22:43:55.231 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_data": 34304, 2020-03-11T22:43:55.231 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_omap": 6, 2020-03-11T22:43:55.231 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_meta": 3145721, 2020-03-11T22:43:55.231 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_avail": 311392448, 2020-03-11T22:43:55.232 INFO:tasks.workunit.client.0.smithi145.stdout: "statfs": { 2020-03-11T22:43:55.232 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 322122547200, 2020-03-11T22:43:55.232 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 318865866752, 2020-03-11T22:43:55.232 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 3221225472, 2020-03-11T22:43:55.232 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 35127296, 2020-03-11T22:43:55.233 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 432213, 2020-03-11T22:43:55.233 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2020-03-11T22:43:55.233 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2020-03-11T22:43:55.233 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2020-03-11T22:43:55.233 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 6856, 2020-03-11T22:43:55.234 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 3221218616 2020-03-11T22:43:55.234 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.234 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_peers": [], 2020-03-11T22:43:55.234 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:43:55.235 INFO:tasks.workunit.client.0.smithi145.stdout: "num_snap_trimming": 0, 2020-03-11T22:43:55.235 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shards_repaired": 0, 2020-03-11T22:43:55.235 INFO:tasks.workunit.client.0.smithi145.stdout: "op_queue_age_hist": { 2020-03-11T22:43:55.235 INFO:tasks.workunit.client.0.smithi145.stdout: "histogram": [], 2020-03-11T22:43:55.235 INFO:tasks.workunit.client.0.smithi145.stdout: "upper_bound": 1 2020-03-11T22:43:55.235 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.235 INFO:tasks.workunit.client.0.smithi145.stdout: "perf_stat": { 2020-03-11T22:43:55.235 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ms": 0, 2020-03-11T22:43:55.236 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ms": 0, 2020-03-11T22:43:55.236 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ns": 0, 2020-03-11T22:43:55.236 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ns": 0 2020-03-11T22:43:55.236 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.236 INFO:tasks.workunit.client.0.smithi145.stdout: "alerts": [], 2020-03-11T22:43:55.236 INFO:tasks.workunit.client.0.smithi145.stdout: "network_ping_times": [] 2020-03-11T22:43:55.236 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:43:55.237 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.237 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_stats": [ 2020-03-11T22:43:55.237 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.237 INFO:tasks.workunit.client.0.smithi145.stdout: "poolid": 1, 2020-03-11T22:43:55.237 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg": 32, 2020-03-11T22:43:55.237 INFO:tasks.workunit.client.0.smithi145.stdout: "stat_sum": { 2020-03-11T22:43:55.237 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes": 48300, 2020-03-11T22:43:55.238 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects": 100, 2020-03-11T22:43:55.238 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_clones": 0, 2020-03-11T22:43:55.238 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_copies": 300, 2020-03-11T22:43:55.238 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:43:55.238 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing": 0, 2020-03-11T22:43:55.238 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_degraded": 100, 2020-03-11T22:43:55.239 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_misplaced": 0, 2020-03-11T22:43:55.239 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_unfound": 0, 2020-03-11T22:43:55.239 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_dirty": 100, 2020-03-11T22:43:55.239 INFO:tasks.workunit.client.0.smithi145.stdout: "num_whiteouts": 0, 2020-03-11T22:43:55.239 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read": 0, 2020-03-11T22:43:55.239 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read_kb": 0, 2020-03-11T22:43:55.239 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write": 100, 2020-03-11T22:43:55.240 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write_kb": 100, 2020-03-11T22:43:55.240 INFO:tasks.workunit.client.0.smithi145.stdout: "num_scrub_errors": 0, 2020-03-11T22:43:55.240 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:43:55.240 INFO:tasks.workunit.client.0.smithi145.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:43:55.240 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_recovered": 0, 2020-03-11T22:43:55.240 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_recovered": 0, 2020-03-11T22:43:55.240 INFO:tasks.workunit.client.0.smithi145.stdout: "num_keys_recovered": 0, 2020-03-11T22:43:55.240 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_omap": 0, 2020-03-11T22:43:55.241 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:43:55.241 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:43:55.241 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush": 0, 2020-03-11T22:43:55.241 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_kb": 0, 2020-03-11T22:43:55.241 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict": 0, 2020-03-11T22:43:55.241 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_kb": 0, 2020-03-11T22:43:55.241 INFO:tasks.workunit.client.0.smithi145.stdout: "num_promote": 0, 2020-03-11T22:43:55.242 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_high": 0, 2020-03-11T22:43:55.242 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_low": 0, 2020-03-11T22:43:55.242 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_some": 0, 2020-03-11T22:43:55.242 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_full": 0, 2020-03-11T22:43:55.242 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_pinned": 0, 2020-03-11T22:43:55.242 INFO:tasks.workunit.client.0.smithi145.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:43:55.242 INFO:tasks.workunit.client.0.smithi145.stdout: "num_large_omap_objects": 0, 2020-03-11T22:43:55.242 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_manifest": 0, 2020-03-11T22:43:55.243 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_bytes": 0, 2020-03-11T22:43:55.243 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_keys": 0, 2020-03-11T22:43:55.243 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_repaired": 0 2020-03-11T22:43:55.243 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.243 INFO:tasks.workunit.client.0.smithi145.stdout: "store_stats": { 2020-03-11T22:43:55.243 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 0, 2020-03-11T22:43:55.243 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 0, 2020-03-11T22:43:55.244 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2020-03-11T22:43:55.244 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 19529728, 2020-03-11T22:43:55.244 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 143228, 2020-03-11T22:43:55.244 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2020-03-11T22:43:55.244 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2020-03-11T22:43:55.244 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2020-03-11T22:43:55.244 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 0, 2020-03-11T22:43:55.245 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 0 2020-03-11T22:43:55.245 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.245 INFO:tasks.workunit.client.0.smithi145.stdout: "log_size": 512, 2020-03-11T22:43:55.245 INFO:tasks.workunit.client.0.smithi145.stdout: "ondisk_log_size": 512, 2020-03-11T22:43:55.245 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 64, 2020-03-11T22:43:55.245 INFO:tasks.workunit.client.0.smithi145.stdout: "acting": 64, 2020-03-11T22:43:55.245 INFO:tasks.workunit.client.0.smithi145.stdout: "num_store_stats": 3 2020-03-11T22:43:55.245 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:43:55.246 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:43:55.246 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_stats": [ 2020-03-11T22:43:55.246 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.246 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2020-03-11T22:43:55.246 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 154618822666 2020-03-11T22:43:55.246 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.246 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.247 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2020-03-11T22:43:55.247 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 42949672975 2020-03-11T22:43:55.247 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.247 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.247 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2020-03-11T22:43:55.247 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 158913789962 2020-03-11T22:43:55.248 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:43:55.248 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:43:55.248 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_by_state": [ 2020-03-11T22:43:55.248 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.248 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "active+undersized+degraded", 2020-03-11T22:43:55.248 INFO:tasks.workunit.client.0.smithi145.stdout: "num": 32 2020-03-11T22:43:55.248 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:43:55.249 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:43:55.249 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_by_osd": [ 2020-03-11T22:43:55.249 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.249 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2020-03-11T22:43:55.249 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 0, 2020-03-11T22:43:55.249 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 32, 2020-03-11T22:43:55.249 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:43:55.249 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.250 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.250 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2020-03-11T22:43:55.250 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 0, 2020-03-11T22:43:55.250 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 0, 2020-03-11T22:43:55.250 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:43:55.250 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.250 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.251 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2020-03-11T22:43:55.251 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 32, 2020-03-11T22:43:55.251 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 32, 2020-03-11T22:43:55.251 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:43:55.251 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:43:55.251 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:43:55.251 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snaps": [ 2020-03-11T22:43:55.252 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:43:55.252 INFO:tasks.workunit.client.0.smithi145.stdout: "pool": 1, 2020-03-11T22:43:55.252 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snaps": {} 2020-03-11T22:43:55.252 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:43:55.252 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:43:55.252 INFO:tasks.workunit.client.0.smithi145.stdout: "servicemap": { 2020-03-11T22:43:55.252 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 1, 2020-03-11T22:43:55.253 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2020-03-11T22:42:15.727805+0000", 2020-03-11T22:43:55.253 INFO:tasks.workunit.client.0.smithi145.stdout: "services": {} 2020-03-11T22:43:55.253 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:43:55.253 INFO:tasks.workunit.client.0.smithi145.stdout: "paxos": { 2020-03-11T22:43:55.253 INFO:tasks.workunit.client.0.smithi145.stdout: "first_committed": 1, 2020-03-11T22:43:55.253 INFO:tasks.workunit.client.0.smithi145.stdout: "last_committed": 320, 2020-03-11T22:43:55.253 INFO:tasks.workunit.client.0.smithi145.stdout: "last_pn": 0, 2020-03-11T22:43:55.254 INFO:tasks.workunit.client.0.smithi145.stdout: "accepted_pn": 0 2020-03-11T22:43:55.254 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:43:55.254 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:43:55.256 INFO:tasks.workunit.client.0.smithi145.stderr:report 2318583219 2020-03-11T22:43:55.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:43:55.256 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:738: TEST_divergent_3: expr 2 + 2 2020-03-11T22:43:55.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:738: TEST_divergent_3: objname=existing_4 2020-03-11T22:43:55.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:739: TEST_divergent_3: echo 'writing non-divergent object existing_4' 2020-03-11T22:43:55.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:740: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:43:55.257 INFO:tasks.workunit.client.0.smithi145.stdout:writing non-divergent object existing_4 2020-03-11T22:43:55.404 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:43:55.405 INFO:tasks.workunit.client.0.smithi145.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.864447+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:43:55.405 INFO:tasks.workunit.client.0.smithi145.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.863935+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:43:55.405 INFO:tasks.workunit.client.0.smithi145.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.863711+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:43:55.405 INFO:tasks.workunit.client.0.smithi145.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.863654+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:43:55.405 INFO:tasks.workunit.client.0.smithi145.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.863301+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:43:55.406 INFO:tasks.workunit.client.0.smithi145.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.861426+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:43:55.406 INFO:tasks.workunit.client.0.smithi145.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.861455+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:43:55.406 INFO:tasks.workunit.client.0.smithi145.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.863162+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:43:55.406 INFO:tasks.workunit.client.0.smithi145.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.863132+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:43:55.406 INFO:tasks.workunit.client.0.smithi145.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.862821+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:43:55.406 INFO:tasks.workunit.client.0.smithi145.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.862377+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:43:55.407 INFO:tasks.workunit.client.0.smithi145.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.862181+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:43:55.407 INFO:tasks.workunit.client.0.smithi145.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.862115+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:43:55.407 INFO:tasks.workunit.client.0.smithi145.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.862609+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:43:55.407 INFO:tasks.workunit.client.0.smithi145.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.862359+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:43:55.407 INFO:tasks.workunit.client.0.smithi145.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.862889+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:43:55.407 INFO:tasks.workunit.client.0.smithi145.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.862616+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:43:55.407 INFO:tasks.workunit.client.0.smithi145.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.861656+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:43:55.408 INFO:tasks.workunit.client.0.smithi145.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.861600+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:43:55.408 INFO:tasks.workunit.client.0.smithi145.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.861699+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:43:55.408 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:35.060971+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:43:55.408 INFO:tasks.workunit.client.0.smithi145.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.863392+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:43:55.408 INFO:tasks.workunit.client.0.smithi145.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.863943+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:43:55.408 INFO:tasks.workunit.client.0.smithi145.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.860484+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:43:55.409 INFO:tasks.workunit.client.0.smithi145.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.860556+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:43:55.409 INFO:tasks.workunit.client.0.smithi145.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.860593+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:43:55.409 INFO:tasks.workunit.client.0.smithi145.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.864446+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:43:55.409 INFO:tasks.workunit.client.0.smithi145.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.861914+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:43:55.409 INFO:tasks.workunit.client.0.smithi145.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.861007+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:43:55.410 INFO:tasks.workunit.client.0.smithi145.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.862087+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:43:55.410 INFO:tasks.workunit.client.0.smithi145.stdout:1.e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.861083+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:43:55.410 INFO:tasks.workunit.client.0.smithi145.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.863450+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:43:55.410 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:43:55.410 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:43:55.411 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:43:55.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: rados -p test put existing_4 /etc/resolv.conf 2020-03-11T22:43:55.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:43:55.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: wait_for_clean 2020-03-11T22:43:55.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:43:55.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:43:55.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:43:55.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:43:55.459 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:43:55.459 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:43:55.459 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:43:55.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:43:55.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:43:55.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:43:55.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:43:55.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:43:55.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:43:55.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:43:55.518 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:43:55.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:43:55.867 INFO:tasks.workunit.client.0.smithi145.stderr:1 2020-03-11T22:43:55.868 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2020-03-11T22:43:55.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:43:55.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:43:55.868 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:43:56.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=154618822669 2020-03-11T22:43:56.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-154618822669' 2020-03-11T22:43:56.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:43:56.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:43:56.107 INFO:tasks.workunit.client.0.smithi145.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:43:56.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:43:56.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-154618822669 1-' 2020-03-11T22:43:56.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:43:56.115 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:43:56.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=158913789964 2020-03-11T22:43:56.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-154618822669 1- 2-158913789964' 2020-03-11T22:43:56.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:43:56.246 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-154618822669 2020-03-11T22:43:56.246 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:43:56.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:43:56.247 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-154618822669 2020-03-11T22:43:56.247 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:43:56.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=154618822669 2020-03-11T22:43:56.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 154618822669' 2020-03-11T22:43:56.248 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 154618822669 2020-03-11T22:43:56.248 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:43:56.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 154618822667 -lt 154618822669 2020-03-11T22:43:56.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:43:57.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:43:57.591 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:43:57.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 154618822669 -lt 154618822669 2020-03-11T22:43:57.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:43:57.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:43:57.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:43:57.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:43:57.947 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:43:57.947 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:43:57.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:43:57.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:43:57.948 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:43:57.948 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 2020-03-11T22:43:58.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672975 -lt 2020-03-11T22:43:58.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672975: unary operator expected 2020-03-11T22:43:58.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:43:58.291 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-158913789964 2020-03-11T22:43:58.291 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:43:58.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:43:58.292 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-158913789964 2020-03-11T22:43:58.292 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:43:58.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=158913789964 2020-03-11T22:43:58.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 158913789964' 2020-03-11T22:43:58.293 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 158913789964 2020-03-11T22:43:58.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:43:58.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 158913789965 -lt 158913789964 2020-03-11T22:43:58.637 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:43:58.638 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:58.638 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:59.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 32 == 0 2020-03-11T22:43:59.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:59.104 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:59.104 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:59.104 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:59.104 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:59.105 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:59.105 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:59.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:43:59.442 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:59.443 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:59.443 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:59.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:43:59.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:43:59.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:43:59.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:43:59.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:43:59.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:43:59.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:43:59.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:59.984 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:59.985 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:59.985 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:59.985 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:59.985 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:59.985 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:00.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:00.316 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:00.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:00.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:00.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:44:00.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:00.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:00.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:00.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:00.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:00.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:00.758 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:00.758 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:01.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:01.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:01.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:44:01.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:01.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:44:01.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:01.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:01.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:01.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:01.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:01.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:01.404 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:01.404 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:01.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:01.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:01.737 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:01.737 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:02.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:44:02.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:02.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:02.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:02.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:02.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:02.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:02.188 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:02.188 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:02.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:02.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:02.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:44:02.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:02.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:44:03.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:03.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:03.030 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:03.031 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:03.031 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:03.031 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:03.031 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:03.031 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:03.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:03.367 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:03.367 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:03.368 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:03.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:44:03.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:03.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:03.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:03.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:03.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:03.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:03.810 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:03.810 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:04.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:04.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:04.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:44:04.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:04.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:44:05.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:05.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:05.056 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:05.056 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:05.056 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:05.056 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:05.056 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:05.057 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:05.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:05.394 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:05.394 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:05.395 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:05.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:44:05.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:05.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:05.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:05.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:05.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:05.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:05.847 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:05.847 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:06.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:06.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:06.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:44:06.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:06.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:44:07.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:07.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:07.892 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:07.892 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:07.893 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:07.893 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:07.893 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:07.893 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:08.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:08.236 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:08.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:08.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:08.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:44:08.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:08.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:08.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:08.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:08.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:08.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:08.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:08.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:09.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:09.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:09.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:44:09.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:09.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:44:12.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:12.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:12.329 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:12.329 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:12.329 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:12.329 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:12.329 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:12.329 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:12.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:12.674 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:12.674 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:12.674 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:13.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:44:13.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:13.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:13.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:13.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:13.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:13.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:13.113 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:13.113 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:13.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:13.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:13.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:44:13.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:13.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:44:19.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:19.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:19.955 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:19.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:19.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:19.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:19.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:19.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:20.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:20.303 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:20.303 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:20.304 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:20.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:44:20.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:20.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:20.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:20.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:20.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:20.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:20.746 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:20.746 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:21.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:21.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:21.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:44:21.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:21.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:44:28.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:28.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:28.492 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:28.492 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:28.492 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:28.492 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:28.492 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:28.492 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:28.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:28.838 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:28.839 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:28.839 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:29.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:44:29.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:29.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:29.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:29.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:29.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:29.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:29.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:29.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:29.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:29.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:29.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:44:29.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:44:30.165 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:44:30.165 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_fingerprint": "9c96a21d-af8e-49c8-8736-1bc757ebabc1", 2020-03-11T22:44:30.165 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:44:30.165 INFO:tasks.workunit.client.0.smithi145.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:44:30.166 INFO:tasks.workunit.client.0.smithi145.stdout: "timestamp": "2020-03-11T22:44:30.161725+0000", 2020-03-11T22:44:30.166 INFO:tasks.workunit.client.0.smithi145.stdout: "tag": "", 2020-03-11T22:44:30.166 INFO:tasks.workunit.client.0.smithi145.stdout: "health": { 2020-03-11T22:44:30.166 INFO:tasks.workunit.client.0.smithi145.stdout: "status": "HEALTH_WARN", 2020-03-11T22:44:30.166 INFO:tasks.workunit.client.0.smithi145.stdout: "checks": { 2020-03-11T22:44:30.166 INFO:tasks.workunit.client.0.smithi145.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:44:30.166 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:44:30.167 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2020-03-11T22:44:30.167 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:44:30.167 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 17 2020-03-11T22:44:30.167 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.167 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [], 2020-03-11T22:44:30.167 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2020-03-11T22:44:30.168 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.168 INFO:tasks.workunit.client.0.smithi145.stdout: "OSD_DOWN": { 2020-03-11T22:44:30.168 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:44:30.168 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2020-03-11T22:44:30.168 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "1 osds down", 2020-03-11T22:44:30.168 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 1 2020-03-11T22:44:30.168 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.169 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2020-03-11T22:44:30.169 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.169 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "osd.1 (root=default,host=smithi145) is down" 2020-03-11T22:44:30.169 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:44:30.169 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:44:30.169 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2020-03-11T22:44:30.170 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.170 INFO:tasks.workunit.client.0.smithi145.stdout: "PG_DEGRADED": { 2020-03-11T22:44:30.170 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:44:30.170 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2020-03-11T22:44:30.171 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 32 pgs degraded, 32 pgs undersized", 2020-03-11T22:44:30.171 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 64 2020-03-11T22:44:30.171 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.171 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2020-03-11T22:44:30.171 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.171 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.0 is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:44:30.171 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.172 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.172 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.1 is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:44:30.172 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.172 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.172 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.2 is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:44:30.172 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.173 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.173 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.3 is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:44:30.173 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.173 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.173 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.4 is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:44:30.173 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.173 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.174 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.5 is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:44:30.174 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.174 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.174 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.6 is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:44:30.174 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.174 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.175 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.7 is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:44:30.175 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.175 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.175 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.8 is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:44:30.175 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.175 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.175 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.9 is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:44:30.176 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.176 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.176 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.a is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:44:30.176 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.176 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.176 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.b is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:44:30.177 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.177 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.177 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.c is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:44:30.177 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.177 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.177 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.d is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:44:30.177 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.178 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.178 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.e is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:44:30.178 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.178 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.178 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.f is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:44:30.178 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.178 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.179 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.10 is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:44:30.179 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.179 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.179 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.11 is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:44:30.179 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.179 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.180 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.12 is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:44:30.180 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.180 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.180 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.13 is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:44:30.180 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.180 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.180 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.14 is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:44:30.181 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.181 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.181 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.15 is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:44:30.181 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.182 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.182 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.16 is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:44:30.182 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.182 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.182 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.17 is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:44:30.182 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.182 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.183 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.18 is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:44:30.183 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.183 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.183 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.19 is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:44:30.183 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.183 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.184 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.1a is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:44:30.184 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.184 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.184 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.1b is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:44:30.184 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.184 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.185 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.1c is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:44:30.185 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.185 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.185 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.1d is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:44:30.185 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.185 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.185 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.1e is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:44:30.186 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.186 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.186 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.1f is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:44:30.186 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:44:30.186 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:44:30.186 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2020-03-11T22:44:30.187 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.187 INFO:tasks.workunit.client.0.smithi145.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:44:30.187 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:44:30.187 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2020-03-11T22:44:30.187 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:44:30.187 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 1 2020-03-11T22:44:30.187 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.188 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2020-03-11T22:44:30.188 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.188 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:44:30.188 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.188 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.188 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:44:30.188 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:44:30.189 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:44:30.189 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2020-03-11T22:44:30.189 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:44:30.189 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.189 INFO:tasks.workunit.client.0.smithi145.stdout: "mutes": [] 2020-03-11T22:44:30.189 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.189 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap_first_committed": 1, 2020-03-11T22:44:30.190 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap_last_committed": 1, 2020-03-11T22:44:30.190 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap": { 2020-03-11T22:44:30.190 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 1, 2020-03-11T22:44:30.190 INFO:tasks.workunit.client.0.smithi145.stdout: "fsid": "e665ac71-d8d3-412f-b706-8f1d72482f80", 2020-03-11T22:44:30.190 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2020-03-11T22:42:15.092934Z", 2020-03-11T22:44:30.190 INFO:tasks.workunit.client.0.smithi145.stdout: "created": "2020-03-11T22:42:15.092934Z", 2020-03-11T22:44:30.191 INFO:tasks.workunit.client.0.smithi145.stdout: "min_mon_release": 15, 2020-03-11T22:44:30.191 INFO:tasks.workunit.client.0.smithi145.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:44:30.191 INFO:tasks.workunit.client.0.smithi145.stdout: "features": { 2020-03-11T22:44:30.191 INFO:tasks.workunit.client.0.smithi145.stdout: "persistent": [ 2020-03-11T22:44:30.192 INFO:tasks.workunit.client.0.smithi145.stdout: "kraken", 2020-03-11T22:44:30.192 INFO:tasks.workunit.client.0.smithi145.stdout: "luminous", 2020-03-11T22:44:30.192 INFO:tasks.workunit.client.0.smithi145.stdout: "mimic", 2020-03-11T22:44:30.192 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap-prune", 2020-03-11T22:44:30.192 INFO:tasks.workunit.client.0.smithi145.stdout: "nautilus", 2020-03-11T22:44:30.193 INFO:tasks.workunit.client.0.smithi145.stdout: "octopus" 2020-03-11T22:44:30.193 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:44:30.193 INFO:tasks.workunit.client.0.smithi145.stdout: "optional": [] 2020-03-11T22:44:30.193 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.193 INFO:tasks.workunit.client.0.smithi145.stdout: "mons": [ 2020-03-11T22:44:30.193 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.193 INFO:tasks.workunit.client.0.smithi145.stdout: "rank": 0, 2020-03-11T22:44:30.194 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "a", 2020-03-11T22:44:30.194 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2020-03-11T22:44:30.194 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:44:30.194 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.194 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:44:30.194 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:44:30.195 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 0 2020-03-11T22:44:30.195 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:44:30.195 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:44:30.195 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.195 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:44:30.195 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:44:30.195 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 0, 2020-03-11T22:44:30.196 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 0 2020-03-11T22:44:30.196 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:44:30.196 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:44:30.196 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.196 INFO:tasks.workunit.client.0.smithi145.stdout: "quorum": [ 2020-03-11T22:44:30.196 INFO:tasks.workunit.client.0.smithi145.stdout: 0 2020-03-11T22:44:30.196 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:44:30.197 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap": { 2020-03-11T22:44:30.197 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 43, 2020-03-11T22:44:30.197 INFO:tasks.workunit.client.0.smithi145.stdout: "fsid": "e665ac71-d8d3-412f-b706-8f1d72482f80", 2020-03-11T22:44:30.197 INFO:tasks.workunit.client.0.smithi145.stdout: "created": "2020-03-11T22:42:15.727479+0000", 2020-03-11T22:44:30.197 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2020-03-11T22:43:21.637157+0000", 2020-03-11T22:44:30.197 INFO:tasks.workunit.client.0.smithi145.stdout: "last_up_change": "2020-03-11T22:43:17.298677+0000", 2020-03-11T22:44:30.197 INFO:tasks.workunit.client.0.smithi145.stdout: "last_in_change": "2020-03-11T22:43:18.717475+0000", 2020-03-11T22:44:30.198 INFO:tasks.workunit.client.0.smithi145.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:44:30.198 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_num": 5800384, 2020-03-11T22:44:30.198 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_set": [ 2020-03-11T22:44:30.198 INFO:tasks.workunit.client.0.smithi145.stdout: "nodown", 2020-03-11T22:44:30.198 INFO:tasks.workunit.client.0.smithi145.stdout: "noin", 2020-03-11T22:44:30.198 INFO:tasks.workunit.client.0.smithi145.stdout: "noout", 2020-03-11T22:44:30.199 INFO:tasks.workunit.client.0.smithi145.stdout: "pglog_hardlimit", 2020-03-11T22:44:30.199 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snapdirs", 2020-03-11T22:44:30.199 INFO:tasks.workunit.client.0.smithi145.stdout: "recovery_deletes", 2020-03-11T22:44:30.199 INFO:tasks.workunit.client.0.smithi145.stdout: "sortbitwise" 2020-03-11T22:44:30.199 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:44:30.199 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_version": 7, 2020-03-11T22:44:30.199 INFO:tasks.workunit.client.0.smithi145.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:44:30.200 INFO:tasks.workunit.client.0.smithi145.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:44:30.200 INFO:tasks.workunit.client.0.smithi145.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:44:30.200 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_snapshot": "", 2020-03-11T22:44:30.200 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_max": 1, 2020-03-11T22:44:30.200 INFO:tasks.workunit.client.0.smithi145.stdout: "max_osd": 3, 2020-03-11T22:44:30.200 INFO:tasks.workunit.client.0.smithi145.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:44:30.200 INFO:tasks.workunit.client.0.smithi145.stdout: "min_compat_client": "jewel", 2020-03-11T22:44:30.201 INFO:tasks.workunit.client.0.smithi145.stdout: "require_osd_release": "octopus", 2020-03-11T22:44:30.201 INFO:tasks.workunit.client.0.smithi145.stdout: "pools": [ 2020-03-11T22:44:30.201 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.201 INFO:tasks.workunit.client.0.smithi145.stdout: "pool": 1, 2020-03-11T22:44:30.201 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_name": "test", 2020-03-11T22:44:30.202 INFO:tasks.workunit.client.0.smithi145.stdout: "create_time": "2020-03-11T22:42:36.437686+0000", 2020-03-11T22:44:30.202 INFO:tasks.workunit.client.0.smithi145.stdout: "flags": 1, 2020-03-11T22:44:30.202 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_names": "hashpspool", 2020-03-11T22:44:30.202 INFO:tasks.workunit.client.0.smithi145.stdout: "type": 1, 2020-03-11T22:44:30.203 INFO:tasks.workunit.client.0.smithi145.stdout: "size": 3, 2020-03-11T22:44:30.203 INFO:tasks.workunit.client.0.smithi145.stdout: "min_size": 2, 2020-03-11T22:44:30.203 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_rule": 0, 2020-03-11T22:44:30.203 INFO:tasks.workunit.client.0.smithi145.stdout: "object_hash": 2, 2020-03-11T22:44:30.203 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_autoscale_mode": "on", 2020-03-11T22:44:30.203 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num": 32, 2020-03-11T22:44:30.203 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_placement_num": 1, 2020-03-11T22:44:30.204 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_placement_num_target": 32, 2020-03-11T22:44:30.204 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num_target": 32, 2020-03-11T22:44:30.204 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num_pending": 32, 2020-03-11T22:44:30.204 INFO:tasks.workunit.client.0.smithi145.stdout: "last_pg_merge_meta": { 2020-03-11T22:44:30.204 INFO:tasks.workunit.client.0.smithi145.stdout: "source_pgid": "0.0", 2020-03-11T22:44:30.204 INFO:tasks.workunit.client.0.smithi145.stdout: "ready_epoch": 0, 2020-03-11T22:44:30.205 INFO:tasks.workunit.client.0.smithi145.stdout: "last_epoch_started": 0, 2020-03-11T22:44:30.205 INFO:tasks.workunit.client.0.smithi145.stdout: "last_epoch_clean": 0, 2020-03-11T22:44:30.205 INFO:tasks.workunit.client.0.smithi145.stdout: "source_version": "0'0", 2020-03-11T22:44:30.205 INFO:tasks.workunit.client.0.smithi145.stdout: "target_version": "0'0" 2020-03-11T22:44:30.205 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.205 INFO:tasks.workunit.client.0.smithi145.stdout: "last_change": "42", 2020-03-11T22:44:30.205 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend": "0", 2020-03-11T22:44:30.206 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:44:30.206 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend_preluminous": "42", 2020-03-11T22:44:30.206 INFO:tasks.workunit.client.0.smithi145.stdout: "auid": 0, 2020-03-11T22:44:30.206 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_mode": "selfmanaged", 2020-03-11T22:44:30.206 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_seq": 0, 2020-03-11T22:44:30.206 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_epoch": 0, 2020-03-11T22:44:30.207 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_snaps": [], 2020-03-11T22:44:30.207 INFO:tasks.workunit.client.0.smithi145.stdout: "removed_snaps": "[]", 2020-03-11T22:44:30.207 INFO:tasks.workunit.client.0.smithi145.stdout: "quota_max_bytes": 0, 2020-03-11T22:44:30.207 INFO:tasks.workunit.client.0.smithi145.stdout: "quota_max_objects": 0, 2020-03-11T22:44:30.207 INFO:tasks.workunit.client.0.smithi145.stdout: "tiers": [], 2020-03-11T22:44:30.207 INFO:tasks.workunit.client.0.smithi145.stdout: "tier_of": -1, 2020-03-11T22:44:30.207 INFO:tasks.workunit.client.0.smithi145.stdout: "read_tier": -1, 2020-03-11T22:44:30.208 INFO:tasks.workunit.client.0.smithi145.stdout: "write_tier": -1, 2020-03-11T22:44:30.208 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_mode": "none", 2020-03-11T22:44:30.208 INFO:tasks.workunit.client.0.smithi145.stdout: "target_max_bytes": 0, 2020-03-11T22:44:30.208 INFO:tasks.workunit.client.0.smithi145.stdout: "target_max_objects": 0, 2020-03-11T22:44:30.208 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:44:30.208 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:44:30.209 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:44:30.209 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_min_flush_age": 0, 2020-03-11T22:44:30.209 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_min_evict_age": 0, 2020-03-11T22:44:30.209 INFO:tasks.workunit.client.0.smithi145.stdout: "erasure_code_profile": "", 2020-03-11T22:44:30.209 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_params": { 2020-03-11T22:44:30.209 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "none" 2020-03-11T22:44:30.210 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.210 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_period": 0, 2020-03-11T22:44:30.210 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_count": 0, 2020-03-11T22:44:30.210 INFO:tasks.workunit.client.0.smithi145.stdout: "use_gmt_hitset": true, 2020-03-11T22:44:30.210 INFO:tasks.workunit.client.0.smithi145.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:44:30.210 INFO:tasks.workunit.client.0.smithi145.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:44:30.210 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:44:30.211 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:44:30.211 INFO:tasks.workunit.client.0.smithi145.stdout: "grade_table": [], 2020-03-11T22:44:30.211 INFO:tasks.workunit.client.0.smithi145.stdout: "stripe_width": 0, 2020-03-11T22:44:30.211 INFO:tasks.workunit.client.0.smithi145.stdout: "expected_num_objects": 0, 2020-03-11T22:44:30.211 INFO:tasks.workunit.client.0.smithi145.stdout: "fast_read": false, 2020-03-11T22:44:30.211 INFO:tasks.workunit.client.0.smithi145.stdout: "options": {}, 2020-03-11T22:44:30.211 INFO:tasks.workunit.client.0.smithi145.stdout: "application_metadata": {} 2020-03-11T22:44:30.212 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:44:30.212 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:44:30.212 INFO:tasks.workunit.client.0.smithi145.stdout: "osds": [ 2020-03-11T22:44:30.212 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.213 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2020-03-11T22:44:30.213 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "25919b39-e67c-442a-a46f-e5d1e5ebdbd6", 2020-03-11T22:44:30.213 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 1, 2020-03-11T22:44:30.213 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2020-03-11T22:44:30.213 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2020-03-11T22:44:30.213 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2020-03-11T22:44:30.213 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 5, 2020-03-11T22:44:30.214 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 30, 2020-03-11T22:44:30.214 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 36, 2020-03-11T22:44:30.214 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 42, 2020-03-11T22:44:30.214 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 31, 2020-03-11T22:44:30.214 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2020-03-11T22:44:30.214 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2020-03-11T22:44:30.214 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:44:30.215 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.215 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:44:30.215 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:44:30.215 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 39271 2020-03-11T22:44:30.215 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.215 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.215 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:44:30.216 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:44:30.216 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 39271 2020-03-11T22:44:30.216 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:44:30.216 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:44:30.216 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.216 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2020-03-11T22:44:30.216 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:44:30.217 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.217 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:44:30.217 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:44:30.217 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 39271 2020-03-11T22:44:30.217 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.217 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.217 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:44:30.218 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:44:30.218 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 39271 2020-03-11T22:44:30.218 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:44:30.218 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:44:30.218 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.218 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2020-03-11T22:44:30.218 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:44:30.219 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.219 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:44:30.219 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:44:30.219 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 39271 2020-03-11T22:44:30.219 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.219 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.220 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:44:30.220 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:44:30.220 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 39271 2020-03-11T22:44:30.220 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:44:30.220 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:44:30.220 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.220 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2020-03-11T22:44:30.221 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:44:30.221 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.221 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:44:30.221 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:44:30.221 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 39271 2020-03-11T22:44:30.221 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.221 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.222 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:44:30.222 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:44:30.222 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 39271 2020-03-11T22:44:30.222 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:44:30.222 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:44:30.223 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.223 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6803/39271", 2020-03-11T22:44:30.223 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6805/39271", 2020-03-11T22:44:30.223 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6809/39271", 2020-03-11T22:44:30.223 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6807/39271", 2020-03-11T22:44:30.224 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2020-03-11T22:44:30.224 INFO:tasks.workunit.client.0.smithi145.stdout: "exists", 2020-03-11T22:44:30.224 INFO:tasks.workunit.client.0.smithi145.stdout: "up" 2020-03-11T22:44:30.224 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:44:30.224 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.224 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.224 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2020-03-11T22:44:30.225 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "72907ff7-85ea-4a4a-b62c-397e4c805acb", 2020-03-11T22:44:30.225 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 0, 2020-03-11T22:44:30.225 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2020-03-11T22:44:30.225 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2020-03-11T22:44:30.225 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2020-03-11T22:44:30.225 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 0, 2020-03-11T22:44:30.226 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 0, 2020-03-11T22:44:30.226 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 10, 2020-03-11T22:44:30.227 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 22, 2020-03-11T22:44:30.227 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 32, 2020-03-11T22:44:30.227 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2020-03-11T22:44:30.227 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2020-03-11T22:44:30.227 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:44:30.228 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.228 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:44:30.228 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:44:30.228 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 34828 2020-03-11T22:44:30.228 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.228 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.229 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:44:30.229 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:44:30.229 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 34828 2020-03-11T22:44:30.229 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:44:30.229 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:44:30.229 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.229 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2020-03-11T22:44:30.230 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:44:30.230 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.230 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:44:30.230 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:44:30.230 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 34828 2020-03-11T22:44:30.230 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.230 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.231 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:44:30.231 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:44:30.231 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 34828 2020-03-11T22:44:30.231 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:44:30.231 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:44:30.232 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.232 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2020-03-11T22:44:30.232 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:44:30.232 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.232 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:44:30.232 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:44:30.233 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 34828 2020-03-11T22:44:30.233 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.233 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.233 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:44:30.233 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:44:30.233 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 34828 2020-03-11T22:44:30.233 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:44:30.233 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:44:30.234 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.234 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2020-03-11T22:44:30.234 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:44:30.234 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.234 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:44:30.234 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:44:30.234 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 34828 2020-03-11T22:44:30.235 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.235 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.235 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:44:30.235 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:44:30.235 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 34828 2020-03-11T22:44:30.235 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:44:30.235 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:44:30.236 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.236 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6811/34828", 2020-03-11T22:44:30.236 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6813/34828", 2020-03-11T22:44:30.236 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6817/34828", 2020-03-11T22:44:30.236 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6815/34828", 2020-03-11T22:44:30.236 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2020-03-11T22:44:30.236 INFO:tasks.workunit.client.0.smithi145.stdout: "exists" 2020-03-11T22:44:30.236 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:44:30.237 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.237 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.237 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2020-03-11T22:44:30.237 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "f1f86676-d3cf-4822-8bec-ab32748439d8", 2020-03-11T22:44:30.237 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 1, 2020-03-11T22:44:30.237 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2020-03-11T22:44:30.238 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2020-03-11T22:44:30.238 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2020-03-11T22:44:30.238 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 15, 2020-03-11T22:44:30.238 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 30, 2020-03-11T22:44:30.238 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 37, 2020-03-11T22:44:30.238 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 0, 2020-03-11T22:44:30.238 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 33, 2020-03-11T22:44:30.239 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2020-03-11T22:44:30.239 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2020-03-11T22:44:30.239 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:44:30.239 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.239 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:44:30.239 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:44:30.239 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 39465 2020-03-11T22:44:30.239 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.240 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.240 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:44:30.240 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:44:30.240 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 39465 2020-03-11T22:44:30.240 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:44:30.240 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:44:30.240 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.241 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2020-03-11T22:44:30.241 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:44:30.241 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.241 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:44:30.241 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:44:30.241 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 39465 2020-03-11T22:44:30.241 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.242 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.242 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:44:30.242 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:44:30.242 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 39465 2020-03-11T22:44:30.242 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:44:30.242 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:44:30.242 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.243 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2020-03-11T22:44:30.243 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:44:30.243 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.243 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:44:30.243 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:44:30.243 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 39465 2020-03-11T22:44:30.243 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.243 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.244 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:44:30.244 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:44:30.244 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 39465 2020-03-11T22:44:30.244 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:44:30.244 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:44:30.244 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.244 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2020-03-11T22:44:30.244 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:44:30.245 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.245 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:44:30.245 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:44:30.245 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 39465 2020-03-11T22:44:30.245 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.245 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.245 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:44:30.246 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:44:30.246 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 39465 2020-03-11T22:44:30.246 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:44:30.246 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:44:30.246 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.246 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6811/39465", 2020-03-11T22:44:30.246 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6813/39465", 2020-03-11T22:44:30.247 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6817/39465", 2020-03-11T22:44:30.247 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6815/39465", 2020-03-11T22:44:30.247 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2020-03-11T22:44:30.247 INFO:tasks.workunit.client.0.smithi145.stdout: "exists", 2020-03-11T22:44:30.247 INFO:tasks.workunit.client.0.smithi145.stdout: "up" 2020-03-11T22:44:30.247 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:44:30.247 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:44:30.248 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:44:30.248 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_xinfo": [ 2020-03-11T22:44:30.248 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.248 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2020-03-11T22:44:30.248 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "2020-03-11T22:43:11.224431+0000", 2020-03-11T22:44:30.248 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2020-03-11T22:44:30.248 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2020-03-11T22:44:30.248 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138292836696063, 2020-03-11T22:44:30.249 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2020-03-11T22:44:30.249 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2020-03-11T22:42:22.238601+0000", 2020-03-11T22:44:30.249 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2020-03-11T22:44:30.249 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.249 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.249 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2020-03-11T22:44:30.249 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "2020-03-11T22:43:11.742976+0000", 2020-03-11T22:44:30.250 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2020-03-11T22:44:30.250 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2020-03-11T22:44:30.250 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138292836696063, 2020-03-11T22:44:30.250 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2020-03-11T22:44:30.250 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2020-03-11T22:42:28.436781+0000", 2020-03-11T22:44:30.250 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2020-03-11T22:44:30.250 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.250 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.251 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2020-03-11T22:44:30.251 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "2020-03-11T22:43:12.241054+0000", 2020-03-11T22:44:30.251 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2020-03-11T22:44:30.251 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2020-03-11T22:44:30.251 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138292836696063, 2020-03-11T22:44:30.251 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2020-03-11T22:44:30.251 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2020-03-11T22:42:34.528134+0000", 2020-03-11T22:44:30.252 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2020-03-11T22:44:30.252 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:44:30.252 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:44:30.252 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_upmap": [], 2020-03-11T22:44:30.252 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_upmap_items": [], 2020-03-11T22:44:30.252 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_temp": [], 2020-03-11T22:44:30.252 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_temp": [], 2020-03-11T22:44:30.252 INFO:tasks.workunit.client.0.smithi145.stdout: "blacklist": {}, 2020-03-11T22:44:30.253 INFO:tasks.workunit.client.0.smithi145.stdout: "erasure_code_profiles": { 2020-03-11T22:44:30.253 INFO:tasks.workunit.client.0.smithi145.stdout: "default": { 2020-03-11T22:44:30.253 INFO:tasks.workunit.client.0.smithi145.stdout: "k": "2", 2020-03-11T22:44:30.253 INFO:tasks.workunit.client.0.smithi145.stdout: "m": "2", 2020-03-11T22:44:30.253 INFO:tasks.workunit.client.0.smithi145.stdout: "plugin": "jerasure", 2020-03-11T22:44:30.253 INFO:tasks.workunit.client.0.smithi145.stdout: "technique": "reed_sol_van" 2020-03-11T22:44:30.253 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:44:30.254 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.254 INFO:tasks.workunit.client.0.smithi145.stdout: "removed_snaps_queue": [], 2020-03-11T22:44:30.254 INFO:tasks.workunit.client.0.smithi145.stdout: "new_removed_snaps": [], 2020-03-11T22:44:30.254 INFO:tasks.workunit.client.0.smithi145.stdout: "new_purged_snaps": [], 2020-03-11T22:44:30.254 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_node_flags": {}, 2020-03-11T22:44:30.254 INFO:tasks.workunit.client.0.smithi145.stdout: "device_class_flags": {} 2020-03-11T22:44:30.255 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.255 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_metadata": [ 2020-03-11T22:44:30.255 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.255 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2020-03-11T22:44:30.255 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2020-03-11T22:44:30.255 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6804/39271,v1:127.0.0.1:6805/39271]", 2020-03-11T22:44:30.256 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2020-03-11T22:44:30.256 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2020-03-11T22:44:30.256 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:44:30.256 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:44:30.256 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:44:30.256 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:44:30.256 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:44:30.257 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:44:30.257 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:44:30.257 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:44:30.257 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:44:30.257 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:44:30.257 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:44:30.257 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:44:30.257 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "octopus", 2020-03-11T22:44:30.258 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:44:30.258 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:44:30.258 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:44:30.258 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2020-03-11T22:44:30.258 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XLDP", 2020-03-11T22:44:30.258 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:44:30.258 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2020-03-11T22:44:30.259 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2020-03-11T22:44:30.259 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:44:30.259 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2020-03-11T22:44:30.259 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6802/39271,v1:127.0.0.1:6803/39271]", 2020-03-11T22:44:30.259 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2020-03-11T22:44:30.259 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/39271,v1:127.0.0.1:6809/39271]", 2020-03-11T22:44:30.259 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/39271,v1:127.0.0.1:6807/39271]", 2020-03-11T22:44:30.260 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2020-03-11T22:44:30.260 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2020-03-11T22:44:30.260 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:44:30.260 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:44:30.260 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2020-03-11T22:44:30.260 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32655820", 2020-03-11T22:44:30.260 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:44:30.261 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:44:30.261 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2020-03-11T22:44:30.261 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:44:30.261 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:44:30.261 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2020-03-11T22:44:30.261 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.261 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.262 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2020-03-11T22:44:30.262 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2020-03-11T22:44:30.262 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6812/34828,v1:127.0.0.1:6813/34828]", 2020-03-11T22:44:30.262 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2020-03-11T22:44:30.262 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2020-03-11T22:44:30.262 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:44:30.263 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:44:30.263 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:44:30.263 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:44:30.263 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:44:30.263 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:44:30.263 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:44:30.263 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:44:30.263 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:44:30.264 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:44:30.264 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:44:30.264 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:44:30.264 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "octopus", 2020-03-11T22:44:30.264 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:44:30.264 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:44:30.264 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:44:30.265 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2020-03-11T22:44:30.265 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XLDP", 2020-03-11T22:44:30.265 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:44:30.265 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2020-03-11T22:44:30.265 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2020-03-11T22:44:30.265 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:44:30.265 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2020-03-11T22:44:30.266 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6810/34828,v1:127.0.0.1:6811/34828]", 2020-03-11T22:44:30.266 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2020-03-11T22:44:30.266 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/34828,v1:127.0.0.1:6817/34828]", 2020-03-11T22:44:30.266 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/34828,v1:127.0.0.1:6815/34828]", 2020-03-11T22:44:30.266 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2020-03-11T22:44:30.266 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2020-03-11T22:44:30.266 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:44:30.267 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:44:30.267 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2020-03-11T22:44:30.267 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32655820", 2020-03-11T22:44:30.267 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:44:30.267 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:44:30.267 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2020-03-11T22:44:30.267 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:44:30.267 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:44:30.268 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2020-03-11T22:44:30.268 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.268 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.268 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2020-03-11T22:44:30.268 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2020-03-11T22:44:30.268 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6812/39465,v1:127.0.0.1:6813/39465]", 2020-03-11T22:44:30.268 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2020-03-11T22:44:30.269 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2020-03-11T22:44:30.269 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:44:30.269 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:44:30.269 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:44:30.269 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:44:30.269 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:44:30.269 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:44:30.270 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:44:30.270 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:44:30.270 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:44:30.270 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:44:30.270 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:44:30.270 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:44:30.270 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "octopus", 2020-03-11T22:44:30.270 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:44:30.271 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:44:30.271 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:44:30.271 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2020-03-11T22:44:30.271 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XLDP", 2020-03-11T22:44:30.271 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:44:30.271 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2020-03-11T22:44:30.271 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2020-03-11T22:44:30.272 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:44:30.272 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2020-03-11T22:44:30.272 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6810/39465,v1:127.0.0.1:6811/39465]", 2020-03-11T22:44:30.272 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2020-03-11T22:44:30.272 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/39465,v1:127.0.0.1:6817/39465]", 2020-03-11T22:44:30.272 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/39465,v1:127.0.0.1:6815/39465]", 2020-03-11T22:44:30.272 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2020-03-11T22:44:30.272 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2020-03-11T22:44:30.273 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:44:30.273 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:44:30.273 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2020-03-11T22:44:30.273 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32655820", 2020-03-11T22:44:30.273 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:44:30.273 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:44:30.273 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2020-03-11T22:44:30.273 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:44:30.274 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:44:30.274 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2020-03-11T22:44:30.274 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:44:30.274 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:44:30.274 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap_first_committed": 1, 2020-03-11T22:44:30.274 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap_last_committed": 43, 2020-03-11T22:44:30.274 INFO:tasks.workunit.client.0.smithi145.stdout: "crushmap": { 2020-03-11T22:44:30.275 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": [ 2020-03-11T22:44:30.275 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.275 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2020-03-11T22:44:30.275 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.0", 2020-03-11T22:44:30.275 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2020-03-11T22:44:30.276 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.276 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.276 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2020-03-11T22:44:30.276 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.1", 2020-03-11T22:44:30.276 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2020-03-11T22:44:30.276 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.276 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.276 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2020-03-11T22:44:30.277 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.2", 2020-03-11T22:44:30.277 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2020-03-11T22:44:30.277 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:44:30.277 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:44:30.277 INFO:tasks.workunit.client.0.smithi145.stdout: "types": [ 2020-03-11T22:44:30.277 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.277 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 0, 2020-03-11T22:44:30.278 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd" 2020-03-11T22:44:30.278 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.278 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.278 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2020-03-11T22:44:30.279 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "host" 2020-03-11T22:44:30.279 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.279 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.279 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 2, 2020-03-11T22:44:30.280 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "chassis" 2020-03-11T22:44:30.280 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.280 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.280 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 3, 2020-03-11T22:44:30.280 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "rack" 2020-03-11T22:44:30.280 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.281 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.281 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 4, 2020-03-11T22:44:30.281 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "row" 2020-03-11T22:44:30.281 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.281 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.281 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 5, 2020-03-11T22:44:30.281 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "pdu" 2020-03-11T22:44:30.282 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.282 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.282 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 6, 2020-03-11T22:44:30.282 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "pod" 2020-03-11T22:44:30.282 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.282 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.282 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 7, 2020-03-11T22:44:30.283 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "room" 2020-03-11T22:44:30.283 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.283 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.283 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 8, 2020-03-11T22:44:30.283 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "datacenter" 2020-03-11T22:44:30.283 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.283 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.284 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 9, 2020-03-11T22:44:30.284 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "zone" 2020-03-11T22:44:30.284 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.284 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.284 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 10, 2020-03-11T22:44:30.284 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "region" 2020-03-11T22:44:30.285 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.285 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.285 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2020-03-11T22:44:30.285 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "root" 2020-03-11T22:44:30.285 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:44:30.285 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:44:30.285 INFO:tasks.workunit.client.0.smithi145.stdout: "buckets": [ 2020-03-11T22:44:30.286 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.286 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -1, 2020-03-11T22:44:30.286 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "default", 2020-03-11T22:44:30.286 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2020-03-11T22:44:30.286 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "root", 2020-03-11T22:44:30.286 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2020-03-11T22:44:30.287 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2020-03-11T22:44:30.287 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2020-03-11T22:44:30.287 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:44:30.287 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.287 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -3, 2020-03-11T22:44:30.287 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2020-03-11T22:44:30.287 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2020-03-11T22:44:30.288 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:44:30.288 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:44:30.288 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.288 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.288 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -2, 2020-03-11T22:44:30.288 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "default~hdd", 2020-03-11T22:44:30.288 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2020-03-11T22:44:30.289 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "root", 2020-03-11T22:44:30.289 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2020-03-11T22:44:30.289 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2020-03-11T22:44:30.289 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2020-03-11T22:44:30.289 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:44:30.289 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.290 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -4, 2020-03-11T22:44:30.290 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2020-03-11T22:44:30.290 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2020-03-11T22:44:30.290 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:44:30.290 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:44:30.290 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.290 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.291 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -3, 2020-03-11T22:44:30.291 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "smithi145", 2020-03-11T22:44:30.291 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2020-03-11T22:44:30.291 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "host", 2020-03-11T22:44:30.291 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2020-03-11T22:44:30.291 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2020-03-11T22:44:30.291 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2020-03-11T22:44:30.292 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:44:30.292 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.292 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2020-03-11T22:44:30.292 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2020-03-11T22:44:30.292 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2020-03-11T22:44:30.292 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.292 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.293 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2020-03-11T22:44:30.293 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2020-03-11T22:44:30.293 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 1 2020-03-11T22:44:30.293 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.293 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.293 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2020-03-11T22:44:30.293 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2020-03-11T22:44:30.294 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 2 2020-03-11T22:44:30.294 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:44:30.294 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:44:30.294 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.294 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.294 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -4, 2020-03-11T22:44:30.295 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "smithi145~hdd", 2020-03-11T22:44:30.295 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2020-03-11T22:44:30.295 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "host", 2020-03-11T22:44:30.295 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2020-03-11T22:44:30.295 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2020-03-11T22:44:30.295 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2020-03-11T22:44:30.295 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:44:30.296 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.296 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2020-03-11T22:44:30.296 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2020-03-11T22:44:30.296 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2020-03-11T22:44:30.296 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.296 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.297 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2020-03-11T22:44:30.297 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2020-03-11T22:44:30.297 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 1 2020-03-11T22:44:30.297 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.297 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.297 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2020-03-11T22:44:30.297 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2020-03-11T22:44:30.298 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 2 2020-03-11T22:44:30.298 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:44:30.298 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:44:30.298 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:44:30.298 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:44:30.298 INFO:tasks.workunit.client.0.smithi145.stdout: "rules": [ 2020-03-11T22:44:30.298 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.298 INFO:tasks.workunit.client.0.smithi145.stdout: "rule_id": 0, 2020-03-11T22:44:30.299 INFO:tasks.workunit.client.0.smithi145.stdout: "rule_name": "replicated_rule", 2020-03-11T22:44:30.299 INFO:tasks.workunit.client.0.smithi145.stdout: "ruleset": 0, 2020-03-11T22:44:30.299 INFO:tasks.workunit.client.0.smithi145.stdout: "type": 1, 2020-03-11T22:44:30.299 INFO:tasks.workunit.client.0.smithi145.stdout: "min_size": 1, 2020-03-11T22:44:30.299 INFO:tasks.workunit.client.0.smithi145.stdout: "max_size": 10, 2020-03-11T22:44:30.299 INFO:tasks.workunit.client.0.smithi145.stdout: "steps": [ 2020-03-11T22:44:30.299 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.300 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "take", 2020-03-11T22:44:30.300 INFO:tasks.workunit.client.0.smithi145.stdout: "item": -1, 2020-03-11T22:44:30.300 INFO:tasks.workunit.client.0.smithi145.stdout: "item_name": "default" 2020-03-11T22:44:30.300 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.300 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.300 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "choose_firstn", 2020-03-11T22:44:30.300 INFO:tasks.workunit.client.0.smithi145.stdout: "num": 0, 2020-03-11T22:44:30.300 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "osd" 2020-03-11T22:44:30.301 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.301 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.301 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "emit" 2020-03-11T22:44:30.301 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:44:30.301 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:44:30.301 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:44:30.301 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:44:30.302 INFO:tasks.workunit.client.0.smithi145.stdout: "tunables": { 2020-03-11T22:44:30.302 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_local_tries": 0, 2020-03-11T22:44:30.302 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:44:30.302 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_total_tries": 50, 2020-03-11T22:44:30.302 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:44:30.302 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:44:30.302 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_stable": 1, 2020-03-11T22:44:30.303 INFO:tasks.workunit.client.0.smithi145.stdout: "straw_calc_version": 1, 2020-03-11T22:44:30.303 INFO:tasks.workunit.client.0.smithi145.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:44:30.303 INFO:tasks.workunit.client.0.smithi145.stdout: "profile": "jewel", 2020-03-11T22:44:30.303 INFO:tasks.workunit.client.0.smithi145.stdout: "optimal_tunables": 1, 2020-03-11T22:44:30.303 INFO:tasks.workunit.client.0.smithi145.stdout: "legacy_tunables": 0, 2020-03-11T22:44:30.303 INFO:tasks.workunit.client.0.smithi145.stdout: "minimum_required_version": "jewel", 2020-03-11T22:44:30.303 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables": 1, 2020-03-11T22:44:30.304 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables2": 1, 2020-03-11T22:44:30.304 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v2_rules": 0, 2020-03-11T22:44:30.304 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables3": 1, 2020-03-11T22:44:30.304 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v3_rules": 0, 2020-03-11T22:44:30.304 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v4_buckets": 1, 2020-03-11T22:44:30.304 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables5": 1, 2020-03-11T22:44:30.304 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v5_rules": 0 2020-03-11T22:44:30.305 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.305 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_args": {} 2020-03-11T22:44:30.305 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.305 INFO:tasks.workunit.client.0.smithi145.stdout: "fsmap": { 2020-03-11T22:44:30.305 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 1, 2020-03-11T22:44:30.305 INFO:tasks.workunit.client.0.smithi145.stdout: "default_fscid": -1, 2020-03-11T22:44:30.305 INFO:tasks.workunit.client.0.smithi145.stdout: "compat": { 2020-03-11T22:44:30.305 INFO:tasks.workunit.client.0.smithi145.stdout: "compat": {}, 2020-03-11T22:44:30.306 INFO:tasks.workunit.client.0.smithi145.stdout: "ro_compat": {}, 2020-03-11T22:44:30.306 INFO:tasks.workunit.client.0.smithi145.stdout: "incompat": { 2020-03-11T22:44:30.306 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_1": "base v0.20", 2020-03-11T22:44:30.306 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:44:30.306 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:44:30.306 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:44:30.306 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:44:30.307 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:44:30.307 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_8": "no anchor table", 2020-03-11T22:44:30.307 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_9": "file layout v2", 2020-03-11T22:44:30.307 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:44:30.307 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:44:30.307 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.307 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_flags": { 2020-03-11T22:44:30.308 INFO:tasks.workunit.client.0.smithi145.stdout: "enable_multiple": false, 2020-03-11T22:44:30.308 INFO:tasks.workunit.client.0.smithi145.stdout: "ever_enabled_multiple": false 2020-03-11T22:44:30.308 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.308 INFO:tasks.workunit.client.0.smithi145.stdout: "standbys": [], 2020-03-11T22:44:30.308 INFO:tasks.workunit.client.0.smithi145.stdout: "filesystems": [] 2020-03-11T22:44:30.308 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.308 INFO:tasks.workunit.client.0.smithi145.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:44:30.309 INFO:tasks.workunit.client.0.smithi145.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:44:30.309 INFO:tasks.workunit.client.0.smithi145.stdout: "auth": { 2020-03-11T22:44:30.309 INFO:tasks.workunit.client.0.smithi145.stdout: "first_committed": 1, 2020-03-11T22:44:30.309 INFO:tasks.workunit.client.0.smithi145.stdout: "last_committed": 5, 2020-03-11T22:44:30.309 INFO:tasks.workunit.client.0.smithi145.stdout: "num_secrets": 3 2020-03-11T22:44:30.309 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.310 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg": 32, 2020-03-11T22:44:30.310 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_active": 32, 2020-03-11T22:44:30.310 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_unknown": 0, 2020-03-11T22:44:30.310 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osd": 3, 2020-03-11T22:44:30.310 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_sum": { 2020-03-11T22:44:30.310 INFO:tasks.workunit.client.0.smithi145.stdout: "stat_sum": { 2020-03-11T22:44:30.310 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes": 47947, 2020-03-11T22:44:30.310 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects": 100, 2020-03-11T22:44:30.311 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_clones": 0, 2020-03-11T22:44:30.311 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_copies": 300, 2020-03-11T22:44:30.311 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:44:30.311 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing": 0, 2020-03-11T22:44:30.311 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_degraded": 100, 2020-03-11T22:44:30.311 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_misplaced": 0, 2020-03-11T22:44:30.311 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_unfound": 0, 2020-03-11T22:44:30.312 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_dirty": 100, 2020-03-11T22:44:30.312 INFO:tasks.workunit.client.0.smithi145.stdout: "num_whiteouts": 0, 2020-03-11T22:44:30.312 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read": 0, 2020-03-11T22:44:30.312 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read_kb": 0, 2020-03-11T22:44:30.312 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write": 101, 2020-03-11T22:44:30.312 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write_kb": 101, 2020-03-11T22:44:30.313 INFO:tasks.workunit.client.0.smithi145.stdout: "num_scrub_errors": 0, 2020-03-11T22:44:30.313 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:44:30.313 INFO:tasks.workunit.client.0.smithi145.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:44:30.313 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_recovered": 0, 2020-03-11T22:44:30.313 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_recovered": 0, 2020-03-11T22:44:30.313 INFO:tasks.workunit.client.0.smithi145.stdout: "num_keys_recovered": 0, 2020-03-11T22:44:30.313 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_omap": 0, 2020-03-11T22:44:30.313 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:44:30.314 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:44:30.314 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush": 0, 2020-03-11T22:44:30.314 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_kb": 0, 2020-03-11T22:44:30.314 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict": 0, 2020-03-11T22:44:30.314 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_kb": 0, 2020-03-11T22:44:30.314 INFO:tasks.workunit.client.0.smithi145.stdout: "num_promote": 0, 2020-03-11T22:44:30.314 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_high": 0, 2020-03-11T22:44:30.315 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_low": 0, 2020-03-11T22:44:30.315 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_some": 0, 2020-03-11T22:44:30.315 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_full": 0, 2020-03-11T22:44:30.315 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_pinned": 0, 2020-03-11T22:44:30.315 INFO:tasks.workunit.client.0.smithi145.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:44:30.315 INFO:tasks.workunit.client.0.smithi145.stdout: "num_large_omap_objects": 0, 2020-03-11T22:44:30.315 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_manifest": 0, 2020-03-11T22:44:30.315 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_bytes": 0, 2020-03-11T22:44:30.316 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_keys": 0, 2020-03-11T22:44:30.316 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_repaired": 0 2020-03-11T22:44:30.316 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.316 INFO:tasks.workunit.client.0.smithi145.stdout: "store_stats": { 2020-03-11T22:44:30.316 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 0, 2020-03-11T22:44:30.316 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 0, 2020-03-11T22:44:30.316 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2020-03-11T22:44:30.316 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 0, 2020-03-11T22:44:30.317 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 0, 2020-03-11T22:44:30.317 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2020-03-11T22:44:30.317 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2020-03-11T22:44:30.317 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2020-03-11T22:44:30.317 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 0, 2020-03-11T22:44:30.317 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 0 2020-03-11T22:44:30.317 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.318 INFO:tasks.workunit.client.0.smithi145.stdout: "log_size": 513, 2020-03-11T22:44:30.318 INFO:tasks.workunit.client.0.smithi145.stdout: "ondisk_log_size": 513, 2020-03-11T22:44:30.318 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 64, 2020-03-11T22:44:30.318 INFO:tasks.workunit.client.0.smithi145.stdout: "acting": 64, 2020-03-11T22:44:30.318 INFO:tasks.workunit.client.0.smithi145.stdout: "num_store_stats": 0 2020-03-11T22:44:30.318 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.318 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_sum": { 2020-03-11T22:44:30.319 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 0, 2020-03-11T22:44:30.319 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 0, 2020-03-11T22:44:30.319 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pgs": 65, 2020-03-11T22:44:30.319 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osds": 3, 2020-03-11T22:44:30.319 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_osds": 3, 2020-03-11T22:44:30.319 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:44:30.320 INFO:tasks.workunit.client.0.smithi145.stdout: "kb": 314572800, 2020-03-11T22:44:30.320 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used": 3180352, 2020-03-11T22:44:30.320 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_data": 34304, 2020-03-11T22:44:30.320 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_omap": 6, 2020-03-11T22:44:30.320 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_meta": 3145721, 2020-03-11T22:44:30.320 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_avail": 311392448, 2020-03-11T22:44:30.320 INFO:tasks.workunit.client.0.smithi145.stdout: "statfs": { 2020-03-11T22:44:30.320 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 322122547200, 2020-03-11T22:44:30.321 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 318865866752, 2020-03-11T22:44:30.321 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 3221225472, 2020-03-11T22:44:30.321 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 35127296, 2020-03-11T22:44:30.321 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 431507, 2020-03-11T22:44:30.321 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2020-03-11T22:44:30.321 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2020-03-11T22:44:30.321 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2020-03-11T22:44:30.322 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 6856, 2020-03-11T22:44:30.322 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 3221218616 2020-03-11T22:44:30.322 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.322 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_peers": [], 2020-03-11T22:44:30.322 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:44:30.322 INFO:tasks.workunit.client.0.smithi145.stdout: "num_snap_trimming": 0, 2020-03-11T22:44:30.322 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shards_repaired": 0, 2020-03-11T22:44:30.322 INFO:tasks.workunit.client.0.smithi145.stdout: "op_queue_age_hist": { 2020-03-11T22:44:30.323 INFO:tasks.workunit.client.0.smithi145.stdout: "histogram": [], 2020-03-11T22:44:30.323 INFO:tasks.workunit.client.0.smithi145.stdout: "upper_bound": 1 2020-03-11T22:44:30.323 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.323 INFO:tasks.workunit.client.0.smithi145.stdout: "perf_stat": { 2020-03-11T22:44:30.323 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ms": 0, 2020-03-11T22:44:30.323 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ms": 0, 2020-03-11T22:44:30.323 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ns": 0, 2020-03-11T22:44:30.324 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ns": 0 2020-03-11T22:44:30.324 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.324 INFO:tasks.workunit.client.0.smithi145.stdout: "alerts": [], 2020-03-11T22:44:30.324 INFO:tasks.workunit.client.0.smithi145.stdout: "network_ping_times": [] 2020-03-11T22:44:30.324 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.324 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_sum_by_class": { 2020-03-11T22:44:30.324 INFO:tasks.workunit.client.0.smithi145.stdout: "hdd": { 2020-03-11T22:44:30.325 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 0, 2020-03-11T22:44:30.325 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 0, 2020-03-11T22:44:30.325 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pgs": 65, 2020-03-11T22:44:30.325 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osds": 3, 2020-03-11T22:44:30.325 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_osds": 3, 2020-03-11T22:44:30.325 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:44:30.325 INFO:tasks.workunit.client.0.smithi145.stdout: "kb": 314572800, 2020-03-11T22:44:30.325 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used": 3180352, 2020-03-11T22:44:30.326 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_data": 34304, 2020-03-11T22:44:30.326 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_omap": 6, 2020-03-11T22:44:30.326 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_meta": 3145721, 2020-03-11T22:44:30.326 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_avail": 311392448, 2020-03-11T22:44:30.326 INFO:tasks.workunit.client.0.smithi145.stdout: "statfs": { 2020-03-11T22:44:30.326 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 322122547200, 2020-03-11T22:44:30.326 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 318865866752, 2020-03-11T22:44:30.327 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 3221225472, 2020-03-11T22:44:30.327 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 35127296, 2020-03-11T22:44:30.327 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 431507, 2020-03-11T22:44:30.327 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2020-03-11T22:44:30.327 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2020-03-11T22:44:30.327 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2020-03-11T22:44:30.327 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 6856, 2020-03-11T22:44:30.328 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 3221218616 2020-03-11T22:44:30.328 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.328 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_peers": [], 2020-03-11T22:44:30.328 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:44:30.328 INFO:tasks.workunit.client.0.smithi145.stdout: "num_snap_trimming": 0, 2020-03-11T22:44:30.328 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shards_repaired": 0, 2020-03-11T22:44:30.328 INFO:tasks.workunit.client.0.smithi145.stdout: "op_queue_age_hist": { 2020-03-11T22:44:30.329 INFO:tasks.workunit.client.0.smithi145.stdout: "histogram": [], 2020-03-11T22:44:30.329 INFO:tasks.workunit.client.0.smithi145.stdout: "upper_bound": 1 2020-03-11T22:44:30.329 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.329 INFO:tasks.workunit.client.0.smithi145.stdout: "perf_stat": { 2020-03-11T22:44:30.329 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ms": 0, 2020-03-11T22:44:30.329 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ms": 0, 2020-03-11T22:44:30.329 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ns": 0, 2020-03-11T22:44:30.330 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ns": 0 2020-03-11T22:44:30.330 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.330 INFO:tasks.workunit.client.0.smithi145.stdout: "alerts": [], 2020-03-11T22:44:30.330 INFO:tasks.workunit.client.0.smithi145.stdout: "network_ping_times": [] 2020-03-11T22:44:30.330 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:44:30.330 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.330 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_stats": [ 2020-03-11T22:44:30.330 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.331 INFO:tasks.workunit.client.0.smithi145.stdout: "poolid": 1, 2020-03-11T22:44:30.331 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg": 32, 2020-03-11T22:44:30.331 INFO:tasks.workunit.client.0.smithi145.stdout: "stat_sum": { 2020-03-11T22:44:30.331 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes": 47947, 2020-03-11T22:44:30.331 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects": 100, 2020-03-11T22:44:30.331 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_clones": 0, 2020-03-11T22:44:30.331 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_copies": 300, 2020-03-11T22:44:30.331 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:44:30.332 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing": 0, 2020-03-11T22:44:30.332 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_degraded": 100, 2020-03-11T22:44:30.332 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_misplaced": 0, 2020-03-11T22:44:30.332 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_unfound": 0, 2020-03-11T22:44:30.332 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_dirty": 100, 2020-03-11T22:44:30.332 INFO:tasks.workunit.client.0.smithi145.stdout: "num_whiteouts": 0, 2020-03-11T22:44:30.332 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read": 0, 2020-03-11T22:44:30.333 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read_kb": 0, 2020-03-11T22:44:30.333 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write": 101, 2020-03-11T22:44:30.333 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write_kb": 101, 2020-03-11T22:44:30.333 INFO:tasks.workunit.client.0.smithi145.stdout: "num_scrub_errors": 0, 2020-03-11T22:44:30.333 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:44:30.333 INFO:tasks.workunit.client.0.smithi145.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:44:30.333 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_recovered": 0, 2020-03-11T22:44:30.334 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_recovered": 0, 2020-03-11T22:44:30.334 INFO:tasks.workunit.client.0.smithi145.stdout: "num_keys_recovered": 0, 2020-03-11T22:44:30.334 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_omap": 0, 2020-03-11T22:44:30.334 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:44:30.334 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:44:30.334 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush": 0, 2020-03-11T22:44:30.334 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_kb": 0, 2020-03-11T22:44:30.335 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict": 0, 2020-03-11T22:44:30.335 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_kb": 0, 2020-03-11T22:44:30.335 INFO:tasks.workunit.client.0.smithi145.stdout: "num_promote": 0, 2020-03-11T22:44:30.335 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_high": 0, 2020-03-11T22:44:30.335 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_low": 0, 2020-03-11T22:44:30.335 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_some": 0, 2020-03-11T22:44:30.335 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_full": 0, 2020-03-11T22:44:30.335 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_pinned": 0, 2020-03-11T22:44:30.336 INFO:tasks.workunit.client.0.smithi145.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:44:30.336 INFO:tasks.workunit.client.0.smithi145.stdout: "num_large_omap_objects": 0, 2020-03-11T22:44:30.336 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_manifest": 0, 2020-03-11T22:44:30.336 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_bytes": 0, 2020-03-11T22:44:30.336 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_keys": 0, 2020-03-11T22:44:30.336 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_repaired": 0 2020-03-11T22:44:30.336 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.337 INFO:tasks.workunit.client.0.smithi145.stdout: "store_stats": { 2020-03-11T22:44:30.337 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 0, 2020-03-11T22:44:30.337 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 0, 2020-03-11T22:44:30.337 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2020-03-11T22:44:30.337 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 19529728, 2020-03-11T22:44:30.337 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 142522, 2020-03-11T22:44:30.337 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2020-03-11T22:44:30.338 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2020-03-11T22:44:30.338 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2020-03-11T22:44:30.338 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 0, 2020-03-11T22:44:30.338 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 0 2020-03-11T22:44:30.338 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.338 INFO:tasks.workunit.client.0.smithi145.stdout: "log_size": 513, 2020-03-11T22:44:30.338 INFO:tasks.workunit.client.0.smithi145.stdout: "ondisk_log_size": 513, 2020-03-11T22:44:30.339 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 64, 2020-03-11T22:44:30.339 INFO:tasks.workunit.client.0.smithi145.stdout: "acting": 64, 2020-03-11T22:44:30.339 INFO:tasks.workunit.client.0.smithi145.stdout: "num_store_stats": 3 2020-03-11T22:44:30.339 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:44:30.339 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:44:30.339 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_stats": [ 2020-03-11T22:44:30.339 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.339 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2020-03-11T22:44:30.340 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 154618822675 2020-03-11T22:44:30.340 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.340 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.340 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2020-03-11T22:44:30.340 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 42949672975 2020-03-11T22:44:30.340 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.340 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.341 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2020-03-11T22:44:30.341 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 158913789971 2020-03-11T22:44:30.341 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:44:30.341 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:44:30.341 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_by_state": [ 2020-03-11T22:44:30.341 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.342 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "active+undersized+degraded", 2020-03-11T22:44:30.342 INFO:tasks.workunit.client.0.smithi145.stdout: "num": 32 2020-03-11T22:44:30.342 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:44:30.342 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:44:30.342 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_by_osd": [ 2020-03-11T22:44:30.342 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.342 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2020-03-11T22:44:30.343 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 0, 2020-03-11T22:44:30.343 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 32, 2020-03-11T22:44:30.343 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:44:30.343 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.343 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.343 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2020-03-11T22:44:30.343 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 0, 2020-03-11T22:44:30.344 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 0, 2020-03-11T22:44:30.344 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:44:30.344 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.344 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.344 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2020-03-11T22:44:30.344 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 32, 2020-03-11T22:44:30.344 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 32, 2020-03-11T22:44:30.345 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:44:30.345 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:44:30.345 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:44:30.345 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snaps": [ 2020-03-11T22:44:30.345 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:44:30.345 INFO:tasks.workunit.client.0.smithi145.stdout: "pool": 1, 2020-03-11T22:44:30.345 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snaps": {} 2020-03-11T22:44:30.345 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:44:30.346 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:44:30.346 INFO:tasks.workunit.client.0.smithi145.stdout: "servicemap": { 2020-03-11T22:44:30.346 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 1, 2020-03-11T22:44:30.346 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2020-03-11T22:42:15.727805+0000", 2020-03-11T22:44:30.346 INFO:tasks.workunit.client.0.smithi145.stdout: "services": {} 2020-03-11T22:44:30.346 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:44:30.346 INFO:tasks.workunit.client.0.smithi145.stdout: "paxos": { 2020-03-11T22:44:30.347 INFO:tasks.workunit.client.0.smithi145.stdout: "first_committed": 1, 2020-03-11T22:44:30.347 INFO:tasks.workunit.client.0.smithi145.stdout: "last_committed": 399, 2020-03-11T22:44:30.347 INFO:tasks.workunit.client.0.smithi145.stdout: "last_pn": 0, 2020-03-11T22:44:30.347 INFO:tasks.workunit.client.0.smithi145.stdout: "accepted_pn": 0 2020-03-11T22:44:30.347 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:44:30.347 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:44:30.348 INFO:tasks.workunit.client.0.smithi145.stderr:report 4194679580 2020-03-11T22:44:30.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:44:30.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:746: TEST_divergent_3: echo 'delay recovery' 2020-03-11T22:44:30.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:44:30.349 INFO:tasks.workunit.client.0.smithi145.stdout:delay recovery 2020-03-11T22:44:30.496 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:44:30.496 INFO:tasks.workunit.client.0.smithi145.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.864447+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:30.496 INFO:tasks.workunit.client.0.smithi145.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.863935+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:30.496 INFO:tasks.workunit.client.0.smithi145.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.863711+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:30.496 INFO:tasks.workunit.client.0.smithi145.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.863654+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:30.496 INFO:tasks.workunit.client.0.smithi145.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.863301+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:30.497 INFO:tasks.workunit.client.0.smithi145.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.861426+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:30.497 INFO:tasks.workunit.client.0.smithi145.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.861455+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:30.497 INFO:tasks.workunit.client.0.smithi145.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.863162+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:30.497 INFO:tasks.workunit.client.0.smithi145.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.863132+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:30.497 INFO:tasks.workunit.client.0.smithi145.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.862821+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:30.497 INFO:tasks.workunit.client.0.smithi145.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.862377+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:30.497 INFO:tasks.workunit.client.0.smithi145.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.862181+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:30.498 INFO:tasks.workunit.client.0.smithi145.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.862115+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:30.498 INFO:tasks.workunit.client.0.smithi145.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.862609+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:30.498 INFO:tasks.workunit.client.0.smithi145.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.862359+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:30.498 INFO:tasks.workunit.client.0.smithi145.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.862889+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:30.498 INFO:tasks.workunit.client.0.smithi145.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.862616+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:30.498 INFO:tasks.workunit.client.0.smithi145.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.861656+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:30.498 INFO:tasks.workunit.client.0.smithi145.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.861600+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:30.498 INFO:tasks.workunit.client.0.smithi145.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.861699+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:30.499 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:35.060971+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:30.499 INFO:tasks.workunit.client.0.smithi145.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.863392+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:30.499 INFO:tasks.workunit.client.0.smithi145.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.863943+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:30.499 INFO:tasks.workunit.client.0.smithi145.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.860484+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:30.499 INFO:tasks.workunit.client.0.smithi145.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.860556+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:30.499 INFO:tasks.workunit.client.0.smithi145.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.860593+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:30.499 INFO:tasks.workunit.client.0.smithi145.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.864446+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:30.500 INFO:tasks.workunit.client.0.smithi145.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.861914+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:30.500 INFO:tasks.workunit.client.0.smithi145.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.861007+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:30.500 INFO:tasks.workunit.client.0.smithi145.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.862087+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:30.500 INFO:tasks.workunit.client.0.smithi145.stdout:1.e 3 0 3 0 0 1156 0 0 17 17 active+undersized+degraded 2020-03-11T22:43:21.861083+0000 43'101 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:30.500 INFO:tasks.workunit.client.0.smithi145.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.863450+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:30.500 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:44:30.500 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:44:30.501 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:44:30.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:748: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:44:30.506 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: get_asok_path osd.0 2020-03-11T22:44:30.506 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:44:30.509 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:44:30.509 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:44:30.509 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:30.509 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:44:30.509 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17402/ceph-osd.0.asok 2020-03-11T22:44:30.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:44:30.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17402/ceph-osd.0.asok set_recovery_delay 100000 2020-03-11T22:44:30.598 INFO:tasks.workunit.client.0.smithi145.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:44:30.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:748: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:44:30.606 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: get_asok_path osd.2 2020-03-11T22:44:30.606 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:44:30.607 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:44:30.607 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:44:30.607 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:30.607 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:44:30.607 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17402/ceph-osd.2.asok 2020-03-11T22:44:30.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:44:30.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17402/ceph-osd.2.asok set_recovery_delay 100000 2020-03-11T22:44:30.697 INFO:tasks.workunit.client.0.smithi145.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:44:30.704 INFO:tasks.workunit.client.0.smithi145.stdout:revive divergent 1 2020-03-11T22:44:30.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:754: TEST_divergent_3: echo 'revive divergent 1' 2020-03-11T22:44:30.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:755: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:44:31.023 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:44:31.023 INFO:tasks.workunit.client.0.smithi145.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.864447+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:31.023 INFO:tasks.workunit.client.0.smithi145.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.863935+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:31.023 INFO:tasks.workunit.client.0.smithi145.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.863711+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:31.024 INFO:tasks.workunit.client.0.smithi145.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.863654+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:31.024 INFO:tasks.workunit.client.0.smithi145.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.863301+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:31.024 INFO:tasks.workunit.client.0.smithi145.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.861426+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:31.024 INFO:tasks.workunit.client.0.smithi145.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.861455+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:31.024 INFO:tasks.workunit.client.0.smithi145.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.863162+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:31.024 INFO:tasks.workunit.client.0.smithi145.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.863132+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:31.024 INFO:tasks.workunit.client.0.smithi145.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.862821+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:31.025 INFO:tasks.workunit.client.0.smithi145.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.862377+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:31.025 INFO:tasks.workunit.client.0.smithi145.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.862181+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:31.025 INFO:tasks.workunit.client.0.smithi145.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.862115+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:31.025 INFO:tasks.workunit.client.0.smithi145.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.862609+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:31.025 INFO:tasks.workunit.client.0.smithi145.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.862359+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:31.025 INFO:tasks.workunit.client.0.smithi145.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.862889+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:31.025 INFO:tasks.workunit.client.0.smithi145.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.862616+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:31.026 INFO:tasks.workunit.client.0.smithi145.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.861656+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:31.026 INFO:tasks.workunit.client.0.smithi145.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.861600+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:31.026 INFO:tasks.workunit.client.0.smithi145.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.861699+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:31.026 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:35.060971+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:31.026 INFO:tasks.workunit.client.0.smithi145.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.863392+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:31.026 INFO:tasks.workunit.client.0.smithi145.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.863943+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:31.027 INFO:tasks.workunit.client.0.smithi145.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.860484+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:31.027 INFO:tasks.workunit.client.0.smithi145.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.860556+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:31.027 INFO:tasks.workunit.client.0.smithi145.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.860593+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:31.027 INFO:tasks.workunit.client.0.smithi145.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.864446+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:31.027 INFO:tasks.workunit.client.0.smithi145.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.861914+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:31.027 INFO:tasks.workunit.client.0.smithi145.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.861007+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:31.027 INFO:tasks.workunit.client.0.smithi145.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.862087+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:31.027 INFO:tasks.workunit.client.0.smithi145.stdout:1.e 3 0 3 0 0 1156 0 0 17 17 active+undersized+degraded 2020-03-11T22:43:21.861083+0000 43'101 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:31.028 INFO:tasks.workunit.client.0.smithi145.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.863450+0000 30'100 43:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:31.028 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:44:31.028 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:44:31.028 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:44:31.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:756: TEST_divergent_3: ceph osd set noup 2020-03-11T22:44:31.703 INFO:tasks.workunit.client.0.smithi145.stderr:noup is set 2020-03-11T22:44:31.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:757: TEST_divergent_3: activate_osd td/divergent-priors 1 2020-03-11T22:44:31.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:44:31.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:44:31.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:44:31.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:44:31.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:44:31.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:44:31.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:44:31.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:44:31.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:44:31.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:44:31.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:44:31.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:44:31.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:44:31.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:44:31.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:44:31.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:44:31.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:44:31.716 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:44:31.716 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:31.716 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:44:31.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:44:31.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:44:31.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:44:31.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:44:31.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:44:31.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:44:31.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:44:31.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:31.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:44:31.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:44:31.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:44:31.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:44:31.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:31.718 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:44:31.718 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2020-03-11T22:44:31.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:44:31.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:44:31.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:44:31.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:44:31.740 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:44:31.737+0000 7f9f48822ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:31.750 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:44:31.748+0000 7f9f48822ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:31.751 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:44:31.749+0000 7f9f48822ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:32.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: return 0 2020-03-11T22:44:32.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: sleep 5 2020-03-11T22:44:32.267 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:44:32.265+0000 7f9f48822ec0 -1 Falling back to public interface 2020-03-11T22:44:33.103 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:44:33.100+0000 7f9f48822ec0 -1 osd.1 30 log_to_monitors {default=true} 2020-03-11T22:44:37.070 INFO:tasks.workunit.client.0.smithi145.stdout:delay recovery divergent 2020-03-11T22:44:37.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:760: TEST_divergent_3: echo 'delay recovery divergent' 2020-03-11T22:44:37.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:761: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:44:37.402 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:44:37.402 INFO:tasks.workunit.client.0.smithi145.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.864447+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:37.402 INFO:tasks.workunit.client.0.smithi145.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.863935+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:37.402 INFO:tasks.workunit.client.0.smithi145.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.863711+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:37.402 INFO:tasks.workunit.client.0.smithi145.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.863654+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:37.402 INFO:tasks.workunit.client.0.smithi145.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.863301+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:37.403 INFO:tasks.workunit.client.0.smithi145.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.861426+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:37.403 INFO:tasks.workunit.client.0.smithi145.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.861455+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:37.403 INFO:tasks.workunit.client.0.smithi145.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.863162+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:37.403 INFO:tasks.workunit.client.0.smithi145.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.863132+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:37.403 INFO:tasks.workunit.client.0.smithi145.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.862821+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:37.403 INFO:tasks.workunit.client.0.smithi145.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.862377+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:37.403 INFO:tasks.workunit.client.0.smithi145.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.862181+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:37.404 INFO:tasks.workunit.client.0.smithi145.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.862115+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:37.404 INFO:tasks.workunit.client.0.smithi145.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.862609+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:37.404 INFO:tasks.workunit.client.0.smithi145.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.862359+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:37.404 INFO:tasks.workunit.client.0.smithi145.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.862889+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:37.404 INFO:tasks.workunit.client.0.smithi145.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.862616+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:37.404 INFO:tasks.workunit.client.0.smithi145.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.861656+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:37.404 INFO:tasks.workunit.client.0.smithi145.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.861600+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:37.405 INFO:tasks.workunit.client.0.smithi145.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.861699+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:37.405 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:35.060971+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:37.405 INFO:tasks.workunit.client.0.smithi145.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.863392+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:37.405 INFO:tasks.workunit.client.0.smithi145.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.863943+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:37.405 INFO:tasks.workunit.client.0.smithi145.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.860484+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:37.405 INFO:tasks.workunit.client.0.smithi145.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.860556+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:37.406 INFO:tasks.workunit.client.0.smithi145.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.860593+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:37.406 INFO:tasks.workunit.client.0.smithi145.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.864446+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:37.406 INFO:tasks.workunit.client.0.smithi145.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.861914+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:37.406 INFO:tasks.workunit.client.0.smithi145.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.861007+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:37.406 INFO:tasks.workunit.client.0.smithi145.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.862087+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:37.406 INFO:tasks.workunit.client.0.smithi145.stdout:1.e 3 0 3 0 0 1156 0 0 17 17 active+undersized+degraded 2020-03-11T22:43:21.861083+0000 43'101 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:37.406 INFO:tasks.workunit.client.0.smithi145.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:43:21.863450+0000 30'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:37.406 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:44:37.407 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:44:37.407 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:44:37.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: get_asok_path osd.1 2020-03-11T22:44:37.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:44:37.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:44:37.413 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:44:37.413 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:37.413 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:44:37.413 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17402/ceph-osd.1.asok 2020-03-11T22:44:37.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:44:37.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17402/ceph-osd.1.asok set_recovery_delay 100000 2020-03-11T22:44:37.503 INFO:tasks.workunit.client.0.smithi145.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:44:37.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:764: TEST_divergent_3: ceph osd unset noup 2020-03-11T22:44:38.367 INFO:tasks.workunit.client.0.smithi145.stderr:noup is unset 2020-03-11T22:44:38.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:766: TEST_divergent_3: wait_for_osd up 0 2020-03-11T22:44:38.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:44:38.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:44:38.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:44:38.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:44:38.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:44:38.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:44:38.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:44:38.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:44:38.377 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:44:38.747 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 36 up_thru 42 down_at 31 last_clean_interval [5,30) [v2:127.0.0.1:6802/39271,v1:127.0.0.1:6803/39271] [v2:127.0.0.1:6804/39271,v1:127.0.0.1:6805/39271] exists,up 25919b39-e67c-442a-a46f-e5d1e5ebdbd6 2020-03-11T22:44:38.748 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:44:38.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:44:38.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:44:38.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:44:38.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:767: TEST_divergent_3: wait_for_osd up 1 2020-03-11T22:44:38.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:44:38.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:44:38.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:44:38.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:44:38.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:44:38.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:44:38.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:44:38.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:44:39.091 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 47 up_thru 47 down_at 32 last_clean_interval [10,30) [v2:127.0.0.1:6818/42040,v1:127.0.0.1:6819/42040] [v2:127.0.0.1:6820/42040,v1:127.0.0.1:6821/42040] exists,up 72907ff7-85ea-4a4a-b62c-397e4c805acb 2020-03-11T22:44:39.092 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:44:39.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:44:39.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:44:39.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:44:39.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:768: TEST_divergent_3: wait_for_osd up 2 2020-03-11T22:44:39.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:44:39.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:44:39.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:44:39.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:44:39.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:44:39.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:44:39.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:44:39.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:44:39.439 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 37 up_thru 0 down_at 33 last_clean_interval [15,30) [v2:127.0.0.1:6810/39465,v1:127.0.0.1:6811/39465] [v2:127.0.0.1:6812/39465,v1:127.0.0.1:6813/39465] exists,up f1f86676-d3cf-4822-8bec-ab32748439d8 2020-03-11T22:44:39.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:44:39.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:44:39.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:44:39.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:770: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:44:39.761 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:44:39.761 INFO:tasks.workunit.client.0.smithi145.stdout:1.1f 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.077881+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:39.761 INFO:tasks.workunit.client.0.smithi145.stdout:1.1e 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.079785+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:39.761 INFO:tasks.workunit.client.0.smithi145.stdout:1.1d 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.078016+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:39.761 INFO:tasks.workunit.client.0.smithi145.stdout:1.1c 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.077922+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:39.761 INFO:tasks.workunit.client.0.smithi145.stdout:1.1b 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.077982+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:39.762 INFO:tasks.workunit.client.0.smithi145.stdout:1.1a 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.082317+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:39.762 INFO:tasks.workunit.client.0.smithi145.stdout:1.19 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.082775+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:39.762 INFO:tasks.workunit.client.0.smithi145.stdout:1.18 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.078897+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:39.762 INFO:tasks.workunit.client.0.smithi145.stdout:1.17 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.078804+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:39.762 INFO:tasks.workunit.client.0.smithi145.stdout:1.16 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.078821+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:39.762 INFO:tasks.workunit.client.0.smithi145.stdout:1.15 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.081050+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:39.763 INFO:tasks.workunit.client.0.smithi145.stdout:1.14 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.082259+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:39.763 INFO:tasks.workunit.client.0.smithi145.stdout:1.13 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.079673+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:39.763 INFO:tasks.workunit.client.0.smithi145.stdout:1.12 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.079954+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:39.763 INFO:tasks.workunit.client.0.smithi145.stdout:1.11 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.079479+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:39.763 INFO:tasks.workunit.client.0.smithi145.stdout:1.10 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.080059+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:39.763 INFO:tasks.workunit.client.0.smithi145.stdout:1.5 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.081354+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:39.763 INFO:tasks.workunit.client.0.smithi145.stdout:1.4 4 0 0 0 0 1458 0 0 16 16 peering 2020-03-11T22:44:38.080271+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:39.764 INFO:tasks.workunit.client.0.smithi145.stdout:1.3 4 0 0 0 0 1458 0 0 16 16 peering 2020-03-11T22:44:38.081344+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:39.764 INFO:tasks.workunit.client.0.smithi145.stdout:1.2 4 0 0 0 0 1458 0 0 16 16 peering 2020-03-11T22:44:38.080120+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:39.764 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.079110+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:39.764 INFO:tasks.workunit.client.0.smithi145.stdout:1.1 4 0 0 0 0 1458 0 0 16 16 peering 2020-03-11T22:44:38.079386+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:39.764 INFO:tasks.workunit.client.0.smithi145.stdout:1.6 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.078695+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:39.764 INFO:tasks.workunit.client.0.smithi145.stdout:1.7 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.081602+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:39.764 INFO:tasks.workunit.client.0.smithi145.stdout:1.8 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.082136+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:39.765 INFO:tasks.workunit.client.0.smithi145.stdout:1.9 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.081647+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:39.765 INFO:tasks.workunit.client.0.smithi145.stdout:1.a 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.078277+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:39.765 INFO:tasks.workunit.client.0.smithi145.stdout:1.b 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.081639+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:39.765 INFO:tasks.workunit.client.0.smithi145.stdout:1.c 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.080982+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:39.765 INFO:tasks.workunit.client.0.smithi145.stdout:1.d 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.080575+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:39.765 INFO:tasks.workunit.client.0.smithi145.stdout:1.e 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.081034+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:39.765 INFO:tasks.workunit.client.0.smithi145.stdout:1.f 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.080464+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:39.766 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:44:39.766 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:44:39.766 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:44:39.770 INFO:tasks.workunit.client.0.smithi145.stdout:wait for peering 2020-03-11T22:44:39.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:771: TEST_divergent_3: echo 'wait for peering' 2020-03-11T22:44:39.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:772: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:44:40.093 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:44:40.093 INFO:tasks.workunit.client.0.smithi145.stdout:1.1f 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.077881+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:40.093 INFO:tasks.workunit.client.0.smithi145.stdout:1.1e 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.079785+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:40.094 INFO:tasks.workunit.client.0.smithi145.stdout:1.1d 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.078016+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:40.094 INFO:tasks.workunit.client.0.smithi145.stdout:1.1c 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.077922+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:40.094 INFO:tasks.workunit.client.0.smithi145.stdout:1.1b 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.077982+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:40.094 INFO:tasks.workunit.client.0.smithi145.stdout:1.1a 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.082317+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:40.094 INFO:tasks.workunit.client.0.smithi145.stdout:1.19 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.082775+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:40.094 INFO:tasks.workunit.client.0.smithi145.stdout:1.18 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.078897+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:40.094 INFO:tasks.workunit.client.0.smithi145.stdout:1.17 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.078804+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:40.095 INFO:tasks.workunit.client.0.smithi145.stdout:1.16 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.078821+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:40.095 INFO:tasks.workunit.client.0.smithi145.stdout:1.15 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.081050+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:40.095 INFO:tasks.workunit.client.0.smithi145.stdout:1.14 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.082259+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:40.095 INFO:tasks.workunit.client.0.smithi145.stdout:1.13 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.079673+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:40.095 INFO:tasks.workunit.client.0.smithi145.stdout:1.12 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.079954+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:40.095 INFO:tasks.workunit.client.0.smithi145.stdout:1.11 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.079479+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:40.095 INFO:tasks.workunit.client.0.smithi145.stdout:1.10 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.080059+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:40.096 INFO:tasks.workunit.client.0.smithi145.stdout:1.5 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.081354+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:40.096 INFO:tasks.workunit.client.0.smithi145.stdout:1.4 4 0 0 0 0 1458 0 0 16 16 peering 2020-03-11T22:44:38.080271+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:40.096 INFO:tasks.workunit.client.0.smithi145.stdout:1.3 4 0 0 0 0 1458 0 0 16 16 peering 2020-03-11T22:44:38.081344+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:40.096 INFO:tasks.workunit.client.0.smithi145.stdout:1.2 4 0 0 0 0 1458 0 0 16 16 peering 2020-03-11T22:44:38.080120+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:40.096 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.079110+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:40.096 INFO:tasks.workunit.client.0.smithi145.stdout:1.1 4 0 0 0 0 1458 0 0 16 16 peering 2020-03-11T22:44:38.079386+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:40.096 INFO:tasks.workunit.client.0.smithi145.stdout:1.6 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.078695+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:40.096 INFO:tasks.workunit.client.0.smithi145.stdout:1.7 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.081602+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:40.097 INFO:tasks.workunit.client.0.smithi145.stdout:1.8 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.082136+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:40.097 INFO:tasks.workunit.client.0.smithi145.stdout:1.9 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.081647+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:40.097 INFO:tasks.workunit.client.0.smithi145.stdout:1.a 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.078277+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:40.097 INFO:tasks.workunit.client.0.smithi145.stdout:1.b 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.081639+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:40.097 INFO:tasks.workunit.client.0.smithi145.stdout:1.c 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.080982+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:40.097 INFO:tasks.workunit.client.0.smithi145.stdout:1.d 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.080575+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:40.097 INFO:tasks.workunit.client.0.smithi145.stdout:1.e 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.081034+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:40.098 INFO:tasks.workunit.client.0.smithi145.stdout:1.f 3 0 0 0 0 1457 0 0 16 16 peering 2020-03-11T22:44:38.080464+0000 30'106 47:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:40.098 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:44:40.098 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:44:40.098 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:44:40.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:773: TEST_divergent_3: rados -p test put foo /etc/fstab 2020-03-11T22:44:49.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: echo 'killing divergent 1' 2020-03-11T22:44:49.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:778: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:44:49.883 INFO:tasks.workunit.client.0.smithi145.stdout:killing divergent 1 2020-03-11T22:44:50.213 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:44:50.214 INFO:tasks.workunit.client.0.smithi145.stdout:1.1f 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:44:38.536526+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:50.214 INFO:tasks.workunit.client.0.smithi145.stdout:1.1e 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:44:38.536418+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:50.214 INFO:tasks.workunit.client.0.smithi145.stdout:1.1d 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:44:38.536470+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:50.214 INFO:tasks.workunit.client.0.smithi145.stdout:1.1c 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:44:38.507558+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:50.214 INFO:tasks.workunit.client.0.smithi145.stdout:1.1b 3 1 1 0 0 1509 0 0 10 10 active+recovery_wait+degraded+wait 2020-03-11T22:44:38.536635+0000 30'100 48:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:50.214 INFO:tasks.workunit.client.0.smithi145.stdout:1.1a 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:44:38.530405+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:50.215 INFO:tasks.workunit.client.0.smithi145.stdout:1.19 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:44:38.534662+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:50.215 INFO:tasks.workunit.client.0.smithi145.stdout:1.18 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:44:38.535900+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:50.215 INFO:tasks.workunit.client.0.smithi145.stdout:1.17 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:44:38.507269+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:50.215 INFO:tasks.workunit.client.0.smithi145.stdout:1.16 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:44:38.536377+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:50.215 INFO:tasks.workunit.client.0.smithi145.stdout:1.15 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:44:38.534746+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:50.215 INFO:tasks.workunit.client.0.smithi145.stdout:1.14 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:44:38.534964+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:50.216 INFO:tasks.workunit.client.0.smithi145.stdout:1.13 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:44:38.535160+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:50.216 INFO:tasks.workunit.client.0.smithi145.stdout:1.12 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:44:38.506965+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:50.216 INFO:tasks.workunit.client.0.smithi145.stdout:1.11 3 1 1 0 0 1509 0 0 16 16 active+recovery_wait+degraded+wait 2020-03-11T22:44:38.535923+0000 30'100 48:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:50.216 INFO:tasks.workunit.client.0.smithi145.stdout:1.10 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:44:38.534992+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:50.216 INFO:tasks.workunit.client.0.smithi145.stdout:1.5 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:44:38.535221+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:50.216 INFO:tasks.workunit.client.0.smithi145.stdout:1.4 4 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:44:38.534520+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:50.216 INFO:tasks.workunit.client.0.smithi145.stdout:1.3 4 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:44:38.506450+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:50.217 INFO:tasks.workunit.client.0.smithi145.stdout:1.2 4 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:44:38.535287+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:50.217 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:44:38.536847+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:50.217 INFO:tasks.workunit.client.0.smithi145.stdout:1.1 4 0 0 0 0 1510 0 0 15 15 active+clean+wait 2020-03-11T22:44:38.535242+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:50.217 INFO:tasks.workunit.client.0.smithi145.stdout:1.6 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:44:38.536239+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:50.217 INFO:tasks.workunit.client.0.smithi145.stdout:1.7 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:44:38.532466+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:50.217 INFO:tasks.workunit.client.0.smithi145.stdout:1.8 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:44:38.506186+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:50.217 INFO:tasks.workunit.client.0.smithi145.stdout:1.9 3 1 1 0 0 1510 0 0 11 11 active+recovering+degraded+wait 2020-03-11T22:44:38.537558+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:50.218 INFO:tasks.workunit.client.0.smithi145.stdout:1.a 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:44:38.537042+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:50.218 INFO:tasks.workunit.client.0.smithi145.stdout:1.b 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:44:38.534499+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:50.218 INFO:tasks.workunit.client.0.smithi145.stdout:1.c 3 0 0 0 0 1510 0 0 14 14 active+clean+wait 2020-03-11T22:44:38.532747+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:50.218 INFO:tasks.workunit.client.0.smithi145.stdout:1.d 3 0 0 0 0 1509 0 0 12 12 active+clean+wait 2020-03-11T22:44:38.506735+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:50.218 INFO:tasks.workunit.client.0.smithi145.stdout:1.e 3 1 1 0 0 1156 0 0 14 14 active+recovery_wait+degraded+wait 2020-03-11T22:44:38.534110+0000 43'101 48:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:50.218 INFO:tasks.workunit.client.0.smithi145.stdout:1.f 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:44:38.536007+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:50.219 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:44:50.219 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:44:50.219 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:44:50.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:779: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd.1 2020-03-11T22:44:50.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:44:50.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:44:50.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:44:50.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:44:50.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:44:50.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:44:50.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:782: TEST_divergent_3: expfile=td/divergent-priors/exp.17402.out 2020-03-11T22:44:50.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:783: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.17402.out 2020-03-11T22:44:50.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:44:50.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:44:50.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-03-11T22:44:50.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:44:50.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-03-11T22:44:50.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.17402.out 2020-03-11T22:44:50.918 INFO:tasks.workunit.client.0.smithi145.stderr:Exporting 1.0 info 1.0( v 30'100 (30'90,30'100] local-lis/les=47/48 n=3 ec=22/22 lis/c=47/47 les/c/f=48/48/0 sis=47 pruub=11.326319574s) 2020-03-11T22:44:50.934 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:023dd3d2:::existing_12:head# 2020-03-11T22:44:50.967 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:04c40a0c:::existing_93:head# 2020-03-11T22:44:51.010 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:064ff5fb:::existing_36:head# 2020-03-11T22:44:51.018 INFO:tasks.workunit.client.0.smithi145.stderr:Export successful 2020-03-11T22:44:51.018 INFO:tasks.workunit.client.0.smithi145.stdout: marking collection for removal 2020-03-11T22:44:51.018 INFO:tasks.workunit.client.0.smithi145.stdout:setting '_remove' omap key 2020-03-11T22:44:51.018 INFO:tasks.workunit.client.0.smithi145.stdout:finish_remove_pgs 1.0_head removing 1.0 2020-03-11T22:44:51.040 INFO:tasks.workunit.client.0.smithi145.stderr:Remove successful 2020-03-11T22:44:51.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:784: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.17402.out 2020-03-11T22:44:51.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:44:51.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:44:51.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-03-11T22:44:51.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:44:51.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-03-11T22:44:51.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.17402.out 2020-03-11T22:44:51.979 INFO:tasks.workunit.client.0.smithi145.stdout:get_pg_num_history pg_num_history pg_num_history(e48 pg_nums {1={22=1,42=32}} deleted_pools ) 2020-03-11T22:44:51.979 INFO:tasks.workunit.client.0.smithi145.stdout:Importing pgid 1.0 2020-03-11T22:44:51.980 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:023dd3d2:::existing_12:head# 2020-03-11T22:44:51.980 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:44:52.016 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:04c40a0c:::existing_93:head# 2020-03-11T22:44:52.016 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:44:52.024 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:064ff5fb:::existing_36:head# 2020-03-11T22:44:52.024 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2020-03-11T22:44:52.032 INFO:tasks.workunit.client.0.smithi145.stdout:write_pg epoch 48 info 1.0( v 30'100 (30'90,30'100] local-lis/les=47/48 n=3 ec=22/22 lis/c=47/47 les/c/f=48/48/0 sis=47 pruub=11.326319574s) 2020-03-11T22:44:52.040 INFO:tasks.workunit.client.0.smithi145.stdout:Import successful 2020-03-11T22:44:52.559 INFO:tasks.workunit.client.0.smithi145.stdout:reviving divergent 1 2020-03-11T22:44:52.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:786: TEST_divergent_3: echo 'reviving divergent 1' 2020-03-11T22:44:52.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:787: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:44:52.888 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:44:52.888 INFO:tasks.workunit.client.0.smithi145.stdout:1.1f 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:44:38.536526+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:52.889 INFO:tasks.workunit.client.0.smithi145.stdout:1.1e 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:44:38.536418+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:52.889 INFO:tasks.workunit.client.0.smithi145.stdout:1.1d 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:44:38.536470+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:52.889 INFO:tasks.workunit.client.0.smithi145.stdout:1.1c 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:44:38.507558+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:52.889 INFO:tasks.workunit.client.0.smithi145.stdout:1.1b 3 1 1 0 0 1509 0 0 10 10 active+recovery_wait+degraded+wait 2020-03-11T22:44:38.536635+0000 30'100 48:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:52.889 INFO:tasks.workunit.client.0.smithi145.stdout:1.1a 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:44:38.530405+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:52.889 INFO:tasks.workunit.client.0.smithi145.stdout:1.19 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:44:38.534662+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:52.890 INFO:tasks.workunit.client.0.smithi145.stdout:1.18 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:44:38.535900+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:52.890 INFO:tasks.workunit.client.0.smithi145.stdout:1.17 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:44:38.507269+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:52.890 INFO:tasks.workunit.client.0.smithi145.stdout:1.16 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:44:38.536377+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:52.890 INFO:tasks.workunit.client.0.smithi145.stdout:1.15 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:44:38.534746+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:52.890 INFO:tasks.workunit.client.0.smithi145.stdout:1.14 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:44:38.534964+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:52.890 INFO:tasks.workunit.client.0.smithi145.stdout:1.13 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:44:38.535160+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:52.890 INFO:tasks.workunit.client.0.smithi145.stdout:1.12 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:44:38.506965+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:52.891 INFO:tasks.workunit.client.0.smithi145.stdout:1.11 3 1 1 0 0 1509 0 0 16 16 active+recovery_wait+degraded+wait 2020-03-11T22:44:38.535923+0000 30'100 48:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:52.891 INFO:tasks.workunit.client.0.smithi145.stdout:1.10 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:44:38.534992+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:52.891 INFO:tasks.workunit.client.0.smithi145.stdout:1.5 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:44:38.535221+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:52.891 INFO:tasks.workunit.client.0.smithi145.stdout:1.4 4 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:44:38.534520+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:52.891 INFO:tasks.workunit.client.0.smithi145.stdout:1.3 4 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:44:38.506450+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:52.891 INFO:tasks.workunit.client.0.smithi145.stdout:1.2 4 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:44:38.535287+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:52.891 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:44:38.536847+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:52.892 INFO:tasks.workunit.client.0.smithi145.stdout:1.1 4 0 0 0 0 1510 0 0 15 15 active+clean+wait 2020-03-11T22:44:38.535242+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:52.892 INFO:tasks.workunit.client.0.smithi145.stdout:1.6 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:44:38.536239+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:52.892 INFO:tasks.workunit.client.0.smithi145.stdout:1.7 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:44:38.532466+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:52.892 INFO:tasks.workunit.client.0.smithi145.stdout:1.8 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:44:38.506186+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:52.892 INFO:tasks.workunit.client.0.smithi145.stdout:1.9 3 1 1 0 0 1510 0 0 11 11 active+recovering+degraded+wait 2020-03-11T22:44:38.537558+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:52.892 INFO:tasks.workunit.client.0.smithi145.stdout:1.a 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:44:38.537042+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:52.892 INFO:tasks.workunit.client.0.smithi145.stdout:1.b 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:44:38.534499+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:52.893 INFO:tasks.workunit.client.0.smithi145.stdout:1.c 3 0 0 0 0 1510 0 0 14 14 active+clean+wait 2020-03-11T22:44:38.532747+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:52.893 INFO:tasks.workunit.client.0.smithi145.stdout:1.d 3 0 0 0 0 1509 0 0 12 12 active+clean+wait 2020-03-11T22:44:38.506735+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:52.893 INFO:tasks.workunit.client.0.smithi145.stdout:1.e 3 1 1 0 0 1156 0 0 14 14 active+recovery_wait+degraded+wait 2020-03-11T22:44:38.534110+0000 43'101 48:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:52.893 INFO:tasks.workunit.client.0.smithi145.stdout:1.f 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:44:38.536007+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:44:52.893 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:44:52.893 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:44:52.894 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:44:52.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:788: TEST_divergent_3: activate_osd td/divergent-priors 1 2020-03-11T22:44:52.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:44:52.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:44:52.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:44:52.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:44:52.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:44:52.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:44:52.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:44:52.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:44:52.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:44:52.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:44:52.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:44:52.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:44:52.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:44:52.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:44:52.903 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:44:52.903 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:44:52.904 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:44:52.904 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:44:52.904 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:52.904 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:44:52.904 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:44:52.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:44:52.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:44:52.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:44:52.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:44:52.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:44:52.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:44:52.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:52.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:44:52.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:44:52.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:44:52.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:44:52.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:52.906 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:44:52.907 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2020-03-11T22:44:52.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:44:52.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:44:52.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:44:52.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:44:52.927 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:44:52.923+0000 7fdd1abfeec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:52.928 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:44:52.925+0000 7fdd1abfeec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:52.930 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:44:52.926+0000 7fdd1abfeec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:53.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-03-11T22:44:53.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:44:53.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:44:53.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:44:53.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:44:53.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:44:53.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:44:53.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:44:53.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:44:53.246 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:44:53.445 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:44:53.442+0000 7fdd1abfeec0 -1 Falling back to public interface 2020-03-11T22:44:53.589 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 47 up_thru 47 down_at 32 last_clean_interval [10,30) [v2:127.0.0.1:6818/42040,v1:127.0.0.1:6819/42040] [v2:127.0.0.1:6820/42040,v1:127.0.0.1:6821/42040] exists,up 72907ff7-85ea-4a4a-b62c-397e4c805acb 2020-03-11T22:44:53.589 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:44:53.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:44:53.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:44:53.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:44:53.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:789: TEST_divergent_3: wait_for_osd up 1 2020-03-11T22:44:53.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:44:53.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:44:53.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:44:53.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:44:53.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:44:53.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:44:53.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:44:53.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:44:53.935 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 47 up_thru 47 down_at 32 last_clean_interval [10,30) [v2:127.0.0.1:6818/42040,v1:127.0.0.1:6819/42040] [v2:127.0.0.1:6820/42040,v1:127.0.0.1:6821/42040] exists,up 72907ff7-85ea-4a4a-b62c-397e4c805acb 2020-03-11T22:44:53.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:44:53.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:44:53.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:44:53.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:791: TEST_divergent_3: sleep 20 2020-03-11T22:44:54.181 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:44:54.178+0000 7fdd1abfeec0 -1 osd.1 48 log_to_monitors {default=true} 2020-03-11T22:45:13.937 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: get_asok_path osd.1 2020-03-11T22:45:13.938 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:45:13.938 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:45:13.939 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:45:13.939 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:13.939 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:45:13.939 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17402/ceph-osd.1.asok 2020-03-11T22:45:13.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:45:13.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17402/ceph-osd.1.asok dump_ops_in_flight 2020-03-11T22:45:14.035 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:45:14.035 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [], 2020-03-11T22:45:14.036 INFO:tasks.workunit.client.0.smithi145.stdout: "num_ops": 0 2020-03-11T22:45:14.036 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:45:14.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:794: TEST_divergent_3: echo 'allowing recovery' 2020-03-11T22:45:14.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:795: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:45:14.043 INFO:tasks.workunit.client.0.smithi145.stdout:allowing recovery 2020-03-11T22:45:14.367 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:45:14.367 INFO:tasks.workunit.client.0.smithi145.stdout:1.1f 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:45:03.968228+0000 30'100 58:151 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:14.367 INFO:tasks.workunit.client.0.smithi145.stdout:1.1e 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:45:02.005813+0000 30'100 56:147 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:14.368 INFO:tasks.workunit.client.0.smithi145.stdout:1.1d 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:44:59.909345+0000 30'100 61:133 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:14.368 INFO:tasks.workunit.client.0.smithi145.stdout:1.1c 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:44:55.004473+0000 30'100 58:146 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:14.368 INFO:tasks.workunit.client.0.smithi145.stdout:1.1b 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:45:03.967835+0000 30'100 58:155 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:14.368 INFO:tasks.workunit.client.0.smithi145.stdout:1.1a 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:45:02.088020+0000 30'100 60:151 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:14.368 INFO:tasks.workunit.client.0.smithi145.stdout:1.19 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:44:59.909355+0000 30'100 61:133 [2,0,1] 2 [2,0,1] 2 30'100 2020-03-11T22:44:58.175003+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:14.368 INFO:tasks.workunit.client.0.smithi145.stdout:1.18 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:44:55.004569+0000 30'100 58:147 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:14.368 INFO:tasks.workunit.client.0.smithi145.stdout:1.17 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:45:03.967502+0000 30'100 58:151 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:14.369 INFO:tasks.workunit.client.0.smithi145.stdout:1.16 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:45:02.088314+0000 30'100 56:147 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:14.369 INFO:tasks.workunit.client.0.smithi145.stdout:1.15 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:44:59.910385+0000 30'100 61:133 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:14.369 INFO:tasks.workunit.client.0.smithi145.stdout:1.14 3 0 0 0 0 1449 0 0 10 10 active+clean 2020-03-11T22:45:08.190141+0000 30'100 59:150 [1,0,2] 1 [1,0,2] 1 30'100 2020-03-11T22:45:08.190095+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:14.369 INFO:tasks.workunit.client.0.smithi145.stdout:1.13 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:45:03.966848+0000 30'100 58:153 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:14.369 INFO:tasks.workunit.client.0.smithi145.stdout:1.12 1 0 0 0 0 483 0 0 16 16 active+clean 2020-03-11T22:45:05.975241+0000 30'100 60:154 [0,1,2] 0 [0,1,2] 0 30'100 2020-03-11T22:45:05.975194+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:14.369 INFO:tasks.workunit.client.0.smithi145.stdout:1.11 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:44:59.908773+0000 30'100 61:133 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:14.369 INFO:tasks.workunit.client.0.smithi145.stdout:1.10 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:44:55.003846+0000 30'100 58:147 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:14.370 INFO:tasks.workunit.client.0.smithi145.stdout:1.5 3 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:44:59.909609+0000 30'100 61:133 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:14.370 INFO:tasks.workunit.client.0.smithi145.stdout:1.4 2 0 0 0 0 966 0 0 16 16 active+clean 2020-03-11T22:45:06.166949+0000 30'100 58:149 [1,0,2] 1 [1,0,2] 1 30'100 2020-03-11T22:45:06.166902+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:14.370 INFO:tasks.workunit.client.0.smithi145.stdout:1.3 4 0 0 0 0 1510 0 0 10 10 active+clean 2020-03-11T22:45:03.967087+0000 30'100 58:153 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:14.370 INFO:tasks.workunit.client.0.smithi145.stdout:1.2 1 0 0 0 0 483 0 0 16 16 active+clean 2020-03-11T22:45:10.118902+0000 30'100 61:154 [0,1,2] 0 [0,1,2] 0 30'100 2020-03-11T22:45:10.118853+0000 30'100 2020-03-11T22:45:10.118853+0000 0 2020-03-11T22:45:14.370 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:44:55.004089+0000 30'100 58:147 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:14.370 INFO:tasks.workunit.client.0.smithi145.stdout:1.1 4 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:44:59.910683+0000 30'100 61:133 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:14.370 INFO:tasks.workunit.client.0.smithi145.stdout:1.6 4 0 0 0 0 1993 0 0 17 17 active+clean 2020-03-11T22:45:02.088287+0000 48'101 56:144 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:14.371 INFO:tasks.workunit.client.0.smithi145.stdout:1.7 3 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:45:03.967474+0000 30'100 58:151 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:14.371 INFO:tasks.workunit.client.0.smithi145.stdout:1.8 3 0 0 0 0 1449 0 0 10 10 active+clean 2020-03-11T22:45:04.093973+0000 30'100 58:150 [1,0,2] 1 [1,0,2] 1 30'100 2020-03-11T22:45:04.093929+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:14.371 INFO:tasks.workunit.client.0.smithi145.stdout:1.9 5 0 0 0 0 2415 0 0 16 16 active+clean 2020-03-11T22:45:11.977191+0000 30'100 61:136 [2,0,1] 2 [2,0,1] 2 30'100 2020-03-11T22:45:11.977145+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:14.371 INFO:tasks.workunit.client.0.smithi145.stdout:1.a 3 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:45:02.087250+0000 30'100 60:151 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:14.371 INFO:tasks.workunit.client.0.smithi145.stdout:1.b 3 0 0 0 0 1510 0 0 10 10 active+clean 2020-03-11T22:45:03.967903+0000 30'100 58:153 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:14.371 INFO:tasks.workunit.client.0.smithi145.stdout:1.c 3 0 0 0 0 1510 0 0 14 14 active+clean 2020-03-11T22:44:55.000998+0000 30'100 58:147 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:14.371 INFO:tasks.workunit.client.0.smithi145.stdout:1.d 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:44:59.909504+0000 30'100 61:133 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:14.372 INFO:tasks.workunit.client.0.smithi145.stdout:1.e 3 0 0 0 0 1156 0 0 17 17 active+clean 2020-03-11T22:45:02.087165+0000 43'101 56:148 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:14.372 INFO:tasks.workunit.client.0.smithi145.stdout:1.f 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:45:03.958007+0000 30'100 58:151 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:14.372 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:45:14.372 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:45:14.372 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:45:14.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2020-03-11T22:45:14.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.0 debug kick_recovery_wq 0 2020-03-11T22:45:14.499 INFO:tasks.workunit.client.0.smithi145.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:45:14.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2020-03-11T22:45:14.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.1 debug kick_recovery_wq 0 2020-03-11T22:45:14.628 INFO:tasks.workunit.client.0.smithi145.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:45:14.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2020-03-11T22:45:14.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.2 debug kick_recovery_wq 0 2020-03-11T22:45:14.756 INFO:tasks.workunit.client.0.smithi145.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:45:14.764 INFO:tasks.workunit.client.0.smithi145.stdout:reading divergent objects 2020-03-11T22:45:14.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:802: TEST_divergent_3: echo 'reading divergent objects' 2020-03-11T22:45:14.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:803: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:45:15.083 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:45:15.083 INFO:tasks.workunit.client.0.smithi145.stdout:1.1f 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:45:03.968228+0000 30'100 58:151 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:15.083 INFO:tasks.workunit.client.0.smithi145.stdout:1.1e 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:45:02.005813+0000 30'100 56:147 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:15.084 INFO:tasks.workunit.client.0.smithi145.stdout:1.1d 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:44:59.909345+0000 30'100 61:133 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:15.084 INFO:tasks.workunit.client.0.smithi145.stdout:1.1c 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:44:55.004473+0000 30'100 58:146 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:15.084 INFO:tasks.workunit.client.0.smithi145.stdout:1.1b 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:45:03.967835+0000 30'100 58:155 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:15.084 INFO:tasks.workunit.client.0.smithi145.stdout:1.1a 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:45:02.088020+0000 30'100 60:151 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:15.084 INFO:tasks.workunit.client.0.smithi145.stdout:1.19 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:44:59.909355+0000 30'100 61:133 [2,0,1] 2 [2,0,1] 2 30'100 2020-03-11T22:44:58.175003+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:15.084 INFO:tasks.workunit.client.0.smithi145.stdout:1.18 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:44:55.004569+0000 30'100 58:147 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:15.084 INFO:tasks.workunit.client.0.smithi145.stdout:1.17 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:45:03.967502+0000 30'100 58:151 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:15.085 INFO:tasks.workunit.client.0.smithi145.stdout:1.16 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:45:02.088314+0000 30'100 56:147 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:15.085 INFO:tasks.workunit.client.0.smithi145.stdout:1.15 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:44:59.910385+0000 30'100 61:133 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:15.085 INFO:tasks.workunit.client.0.smithi145.stdout:1.14 3 0 0 0 0 1449 0 0 10 10 active+clean 2020-03-11T22:45:08.190141+0000 30'100 59:150 [1,0,2] 1 [1,0,2] 1 30'100 2020-03-11T22:45:08.190095+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:15.085 INFO:tasks.workunit.client.0.smithi145.stdout:1.13 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:45:03.966848+0000 30'100 58:153 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:15.085 INFO:tasks.workunit.client.0.smithi145.stdout:1.12 1 0 0 0 0 483 0 0 16 16 active+clean 2020-03-11T22:45:05.975241+0000 30'100 60:154 [0,1,2] 0 [0,1,2] 0 30'100 2020-03-11T22:45:05.975194+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:15.086 INFO:tasks.workunit.client.0.smithi145.stdout:1.11 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:44:59.908773+0000 30'100 61:133 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:15.086 INFO:tasks.workunit.client.0.smithi145.stdout:1.10 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:44:55.003846+0000 30'100 58:147 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:15.086 INFO:tasks.workunit.client.0.smithi145.stdout:1.5 3 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:44:59.909609+0000 30'100 61:133 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:15.086 INFO:tasks.workunit.client.0.smithi145.stdout:1.4 2 0 0 0 0 966 0 0 16 16 active+clean 2020-03-11T22:45:06.166949+0000 30'100 58:149 [1,0,2] 1 [1,0,2] 1 30'100 2020-03-11T22:45:06.166902+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:15.086 INFO:tasks.workunit.client.0.smithi145.stdout:1.3 4 0 0 0 0 1510 0 0 10 10 active+clean 2020-03-11T22:45:03.967087+0000 30'100 58:153 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:15.086 INFO:tasks.workunit.client.0.smithi145.stdout:1.2 1 0 0 0 0 483 0 0 16 16 active+clean 2020-03-11T22:45:10.118902+0000 30'100 61:154 [0,1,2] 0 [0,1,2] 0 30'100 2020-03-11T22:45:10.118853+0000 30'100 2020-03-11T22:45:10.118853+0000 0 2020-03-11T22:45:15.086 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:44:55.004089+0000 30'100 58:147 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:15.087 INFO:tasks.workunit.client.0.smithi145.stdout:1.1 4 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:44:59.910683+0000 30'100 61:133 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:15.087 INFO:tasks.workunit.client.0.smithi145.stdout:1.6 4 0 0 0 0 1993 0 0 17 17 active+clean 2020-03-11T22:45:02.088287+0000 48'101 56:144 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:15.087 INFO:tasks.workunit.client.0.smithi145.stdout:1.7 3 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:45:03.967474+0000 30'100 58:151 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:15.087 INFO:tasks.workunit.client.0.smithi145.stdout:1.8 3 0 0 0 0 1449 0 0 10 10 active+clean 2020-03-11T22:45:04.093973+0000 30'100 58:150 [1,0,2] 1 [1,0,2] 1 30'100 2020-03-11T22:45:04.093929+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:15.087 INFO:tasks.workunit.client.0.smithi145.stdout:1.9 5 0 0 0 0 2415 0 0 16 16 active+clean 2020-03-11T22:45:11.977191+0000 30'100 61:136 [2,0,1] 2 [2,0,1] 2 30'100 2020-03-11T22:45:11.977145+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:15.087 INFO:tasks.workunit.client.0.smithi145.stdout:1.a 3 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:45:02.087250+0000 30'100 60:151 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:15.087 INFO:tasks.workunit.client.0.smithi145.stdout:1.b 3 0 0 0 0 1510 0 0 10 10 active+clean 2020-03-11T22:45:03.967903+0000 30'100 58:153 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:15.088 INFO:tasks.workunit.client.0.smithi145.stdout:1.c 3 0 0 0 0 1510 0 0 14 14 active+clean 2020-03-11T22:44:55.000998+0000 30'100 58:147 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:15.088 INFO:tasks.workunit.client.0.smithi145.stdout:1.d 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:44:59.909504+0000 30'100 61:133 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:15.088 INFO:tasks.workunit.client.0.smithi145.stdout:1.e 3 0 0 0 0 1156 0 0 17 17 active+clean 2020-03-11T22:45:02.087165+0000 43'101 56:148 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:15.088 INFO:tasks.workunit.client.0.smithi145.stdout:1.f 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:45:03.958007+0000 30'100 58:151 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:42:36.520551+0000 0'0 2020-03-11T22:42:36.520551+0000 0 2020-03-11T22:45:15.088 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:45:15.088 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:45:15.089 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:45:15.093 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: expr 2 + 2 2020-03-11T22:45:15.094 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: seq 1 4 2020-03-11T22:45:15.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:45:15.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_1 td/divergent-priors/existing 2020-03-11T22:45:15.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:45:15.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_2 td/divergent-priors/existing 2020-03-11T22:45:15.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:45:15.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_3 td/divergent-priors/existing 2020-03-11T22:45:15.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:45:15.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_4 td/divergent-priors/existing 2020-03-11T22:45:15.243 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: seq 1 2 2020-03-11T22:45:15.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:45:15.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: rados -p test get newobject_1 td/divergent-priors/existing 2020-03-11T22:45:15.268 INFO:tasks.workunit.client.0.smithi145.stderr:error getting test/newobject_1: (2) No such file or directory 2020-03-11T22:45:15.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:45:15.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: rados -p test get newobject_2 td/divergent-priors/existing 2020-03-11T22:45:15.301 INFO:tasks.workunit.client.0.smithi145.stderr:error getting test/newobject_2: (2) No such file or directory 2020-03-11T22:45:15.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: rm -f td/divergent-priors/existing 2020-03-11T22:45:15.301 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2020-03-11T22:45:15.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:45:15.307 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.210+0000 7f9f249d9700 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 30'103 (30'2) modify 1:97db2a0f:::existing_2:head by client.4540.0:1 2020-03-11T22:43:00.512497+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:45:15.307 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.210+0000 7f9f249d9700 20 _merge_object_divergent_entries: keeping 30'103 (30'2) modify 1:97db2a0f:::existing_2:head by client.4540.0:1 2020-03-11T22:43:00.512497+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:45:15.308 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.210+0000 7f9f249d9700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:45:15.308 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.210+0000 7f9f249d9700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 30'2 first_divergent_update: 30'103 last_divergent_update: 30'103 2020-03-11T22:45:15.308 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.210+0000 7f9f249d9700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2020-03-11T22:45:15.308 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.210+0000 7f9f249d9700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:45:15.308 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.210+0000 7f9f249d9700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 30'106 2020-03-11T22:45:15.308 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.210+0000 7f9f249d9700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 30'103 (30'2) modify 1:97db2a0f:::existing_2:head by client.4540.0:1 2020-03-11T22:43:00.512497+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:45:15.308 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.210+0000 7f9f249d9700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2020-03-11T22:45:15.309 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.210+0000 7f9f249d9700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 30'2 <= info.log_tail 30'89 2020-03-11T22:45:15.309 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.288+0000 7f9f249d9700 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 30'106 (30'4) delete 1:70b3bf12:::existing_4:head by client.4545.0:1 2020-03-11T22:43:00.518002+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:45:15.309 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.288+0000 7f9f249d9700 20 _merge_object_divergent_entries: keeping 30'106 (30'4) delete 1:70b3bf12:::existing_4:head by client.4545.0:1 2020-03-11T22:43:00.518002+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:45:15.309 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.288+0000 7f9f249d9700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:45:15.309 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.288+0000 7f9f249d9700 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 30'4 first_divergent_update: 30'106 last_divergent_update: 30'106 2020-03-11T22:45:15.309 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.288+0000 7f9f249d9700 10 _merge_object_divergent_entries: more recent entry found: 43'101 (30'4) modify 1:70b3bf12:::existing_4:head by client.4658.0:1 2020-03-11T22:43:55.438693+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0, already merged 2020-03-11T22:45:15.309 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.288+0000 7f9f249d9700 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 30'101 (0'0) modify 1:75ac5091:::newobject_2:head by client.4532.0:1 2020-03-11T22:43:00.497401+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:45:15.310 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.288+0000 7f9f249d9700 20 _merge_object_divergent_entries: keeping 30'101 (0'0) modify 1:75ac5091:::newobject_2:head by client.4532.0:1 2020-03-11T22:43:00.497401+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:45:15.310 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.288+0000 7f9f249d9700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:45:15.310 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.288+0000 7f9f249d9700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 30'101 last_divergent_update: 30'101 2020-03-11T22:45:15.310 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.288+0000 7f9f249d9700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2020-03-11T22:45:15.310 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.288+0000 7f9f249d9700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2020-03-11T22:45:15.310 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.292+0000 7f9f259db700 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 30'105 (30'3) delete 1:8881e6d8:::existing_3:head by client.4543.0:1 2020-03-11T22:43:00.516036+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:45:15.310 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.292+0000 7f9f259db700 20 _merge_object_divergent_entries: keeping 30'105 (30'3) delete 1:8881e6d8:::existing_3:head by client.4543.0:1 2020-03-11T22:43:00.516036+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:45:15.311 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.292+0000 7f9f259db700 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2020-03-11T22:45:15.311 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.292+0000 7f9f259db700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 30'3 first_divergent_update: 30'105 last_divergent_update: 30'105 2020-03-11T22:45:15.311 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.292+0000 7f9f259db700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2020-03-11T22:45:15.311 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.292+0000 7f9f259db700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:45:15.311 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.292+0000 7f9f259db700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 30'106 2020-03-11T22:45:15.311 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.292+0000 7f9f259db700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 30'105 (30'3) delete 1:8881e6d8:::existing_3:head by client.4543.0:1 2020-03-11T22:43:00.516036+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:45:15.311 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.292+0000 7f9f259db700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2020-03-11T22:45:15.311 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.292+0000 7f9f259db700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 30'3 <= info.log_tail 30'84 2020-03-11T22:45:15.312 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.294+0000 7f9f259db700 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 30'104 (30'1) modify 1:d858c035:::existing_1:head by client.4539.0:1 2020-03-11T22:43:00.512558+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:45:15.312 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.294+0000 7f9f259db700 20 _merge_object_divergent_entries: keeping 30'104 (30'1) modify 1:d858c035:::existing_1:head by client.4539.0:1 2020-03-11T22:43:00.512558+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:45:15.312 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.294+0000 7f9f259db700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:45:15.315 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.294+0000 7f9f259db700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 30'1 first_divergent_update: 30'104 last_divergent_update: 30'104 2020-03-11T22:45:15.315 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.294+0000 7f9f259db700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2020-03-11T22:45:15.315 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.294+0000 7f9f259db700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:45:15.315 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.294+0000 7f9f259db700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 30'106 2020-03-11T22:45:15.315 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.294+0000 7f9f259db700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 30'104 (30'1) modify 1:d858c035:::existing_1:head by client.4539.0:1 2020-03-11T22:43:00.512558+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:45:15.315 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.294+0000 7f9f259db700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2020-03-11T22:45:15.316 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.294+0000 7f9f259db700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 30'1 <= info.log_tail 30'90 2020-03-11T22:45:15.316 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.294+0000 7f9f259db700 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 30'102 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4533.0:1 2020-03-11T22:43:00.498995+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:45:15.316 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.294+0000 7f9f259db700 20 _merge_object_divergent_entries: keeping 30'102 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4533.0:1 2020-03-11T22:43:00.498995+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:45:15.316 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.294+0000 7f9f259db700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:45:15.316 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.294+0000 7f9f259db700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 30'102 last_divergent_update: 30'102 2020-03-11T22:45:15.316 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.294+0000 7f9f259db700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2020-03-11T22:45:15.316 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:44:38.294+0000 7f9f259db700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2020-03-11T22:45:15.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:816: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2020-03-11T22:45:15.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:816: TEST_divergent_3: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:45:15.324 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:822: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2020-03-11T22:45:15.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:822: TEST_divergent_3: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:45:15.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:827: TEST_divergent_3: echo success 2020-03-11T22:45:15.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: rm td/divergent-priors/td/divergent-priors/exp.17402.out 2020-03-11T22:45:15.333 INFO:tasks.workunit.client.0.smithi145.stdout:success 2020-03-11T22:45:15.333 INFO:tasks.workunit.client.0.smithi145.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.17402.out': No such file or directory 2020-03-11T22:45:15.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: delete_pool test 2020-03-11T22:45:15.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test 2020-03-11T22:45:15.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-03-11T22:45:15.985 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' does not exist 2020-03-11T22:45:15.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:832: TEST_divergent_3: kill_daemons td/divergent-priors 2020-03-11T22:45:15.994 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:45:15.994 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:45:15.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:45:15.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:45:15.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:45:16.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:45:16.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2020-03-11T22:45:16.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:45:16.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:45:16.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:45:16.110 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:45:16.111 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:45:16.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:45:16.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:45:16.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:45:16.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:45:16.115 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:45:16.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:45:16.116 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:45:16.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:45:16.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:45:16.117 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:45:16.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:45:16.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:45:16.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:45:16.119 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:45:16.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:45:16.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:45:16.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:45:16.137 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:45:16.138 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:16.138 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:45:16.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17402 2020-03-11T22:45:16.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:45:16.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:45:16.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2020-03-11T22:45:16.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2020-03-11T22:45:16.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2020-03-11T22:45:16.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2020-03-11T22:45:16.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:45:16.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:45:16.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:45:16.141 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:45:16.141 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:45:16.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:45:16.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:45:16.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:45:16.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:45:16.142 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:45:16.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:45:16.143 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:45:16.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:45:16.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:45:16.143 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:45:16.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:45:16.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:45:16.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:45:16.146 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:45:16.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:45:16.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:45:16.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:45:16.148 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:45:16.148 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:16.148 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:45:16.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17402 2020-03-11T22:45:16.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:45:16.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:45:16.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2020-03-11T22:45:16.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:45:16.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:16.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:45:16.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17402 2020-03-11T22:45:16.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:45:16.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-03-11T22:45:16.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:45:16.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2020-03-11T22:45:16.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_ec td/divergent-priors 2020-03-11T22:45:16.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:228: TEST_divergent_ec: local dir=td/divergent-priors 2020-03-11T22:45:16.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:231: TEST_divergent_ec: local dummyfile=/etc/fstab 2020-03-11T22:45:16.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:232: TEST_divergent_ec: local dummyfile2=/etc/resolv.conf 2020-03-11T22:45:16.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:234: TEST_divergent_ec: local num_osds=3 2020-03-11T22:45:16.154 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: expr 3 - 1 2020-03-11T22:45:16.154 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: seq 0 2 2020-03-11T22:45:16.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: local 'osds=0 2020-03-11T22:45:16.155 INFO:tasks.workunit.client.0.smithi145.stderr:1 2020-03-11T22:45:16.155 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2020-03-11T22:45:16.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:236: TEST_divergent_ec: run_mon td/divergent-priors a 2020-03-11T22:45:16.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2020-03-11T22:45:16.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:45:16.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:45:16.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:45:16.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2020-03-11T22:45:16.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2020-03-11T22:45:16.456 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:45:16.456 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:45:16.456 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:45:16.456 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:45:16.456 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:16.457 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:45:16.457 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:45:16.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:45:16.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:45:16.619 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:45:16.619 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:45:16.619 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:45:16.619 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:45:16.619 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:45:16.620 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:45:16.620 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:45:16.620 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:45:16.620 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:45:16.620 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:16.620 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:45:16.620 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17402/ceph-mon.a.asok 2020-03-11T22:45:16.621 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:45:16.621 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17402/ceph-mon.a.asok config get fsid 2020-03-11T22:45:16.704 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:45:16.704 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:45:16.704 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:45:16.704 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:45:16.704 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:45:16.705 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:45:16.705 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:45:16.705 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:45:16.705 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:45:16.705 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:16.705 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:45:16.705 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17402/ceph-mon.a.asok 2020-03-11T22:45:16.706 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:45:16.706 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17402/ceph-mon.a.asok config get mon_host 2020-03-11T22:45:16.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:237: TEST_divergent_ec: run_mgr td/divergent-priors x 2020-03-11T22:45:16.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/divergent-priors 2020-03-11T22:45:16.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:45:16.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:45:16.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:45:16.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/divergent-priors/x 2020-03-11T22:45:16.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:45:17.673 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:45:17.673 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:45:17.673 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:45:17.673 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:45:17.674 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:17.674 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:45:17.674 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:45:17.674 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:45:17.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:45:17.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2020-03-11T22:45:17.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 0 2020-03-11T22:45:17.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:45:17.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:45:17.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:45:17.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:45:17.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:45:17.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:45:17.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:45:17.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:45:17.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:45:17.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:45:17.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:45:17.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:45:17.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:45:17.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:45:17.704 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:45:17.704 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:45:17.704 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:45:17.704 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:45:17.704 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:17.705 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:45:17.705 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:45:17.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:45:17.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:45:17.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:45:17.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:45:17.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:45:17.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:45:17.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:45:17.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:45:17.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:17.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:45:17.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:45:17.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:45:17.707 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:45:17.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=04216e1d-6638-4e33-8cad-1ea8a9c30138 2020-03-11T22:45:17.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 04216e1d-6638-4e33-8cad-1ea8a9c30138' 2020-03-11T22:45:17.708 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:45:17.708 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 04216e1d-6638-4e33-8cad-1ea8a9c30138 2020-03-11T22:45:17.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQD9aWle/fPVKhAAKzk691jYIu60zWyzOcewkg== 2020-03-11T22:45:17.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQD9aWle/fPVKhAAKzk691jYIu60zWyzOcewkg=="}' 2020-03-11T22:45:17.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 04216e1d-6638-4e33-8cad-1ea8a9c30138 -i td/divergent-priors/0/new.json 2020-03-11T22:45:18.070 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:45:18.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/0/new.json 2020-03-11T22:45:18.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD9aWle/fPVKhAAKzk691jYIu60zWyzOcewkg== --osd-uuid 04216e1d-6638-4e33-8cad-1ea8a9c30138 2020-03-11T22:45:18.109 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:45:18.106+0000 7f728acbaec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:18.110 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:45:18.108+0000 7f728acbaec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:18.111 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:45:18.109+0000 7f728acbaec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:18.112 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:45:18.109+0000 7f728acbaec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:45:18.112 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:45:18.109+0000 7f728acbaec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:45:18.112 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:45:18.109+0000 7f728acbaec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:45:18.156 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:45:18.153+0000 7f728acbaec0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2020-03-11T22:45:21.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/0/keyring 2020-03-11T22:45:21.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:45:21.084 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2020-03-11T22:45:21.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:45:21.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:45:21.548 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2020-03-11T22:45:21.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:45:21.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:21.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:45:21.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:45:21.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:45:21.574 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:45:21.560+0000 7fcdfb962ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:21.575 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:45:21.562+0000 7fcdfb962ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:21.576 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:45:21.564+0000 7fcdfb962ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:21.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:45:21.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:45:21.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:45:21.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:45:21.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:45:21.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:21.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:45:21.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:21.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:45:21.885 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:45:22.082 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:45:22.078+0000 7fcdfb962ec0 -1 Falling back to public interface 2020-03-11T22:45:22.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:45:22.855 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:45:22.852+0000 7fcdfb962ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:45:23.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:45:23.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:23.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:45:23.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:23.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:45:23.246 INFO:tasks.workunit.client.0.smithi145.stdout:1 2020-03-11T22:45:23.655 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/43536,v1:127.0.0.1:6803/43536] [v2:127.0.0.1:6804/43536,v1:127.0.0.1:6805/43536] exists,up 04216e1d-6638-4e33-8cad-1ea8a9c30138 2020-03-11T22:45:23.655 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 bc68bb65-33a0-4802-9e9b-c0869e7c2f4b 2020-03-11T22:45:23.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:45:23.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:45:23.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:45:23.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2020-03-11T22:45:23.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 1 2020-03-11T22:45:23.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:45:23.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:45:23.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:45:23.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:45:23.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:45:23.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:45:23.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:45:23.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:45:23.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:45:23.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:45:23.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:45:23.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:45:23.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:45:23.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:45:23.659 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:45:23.659 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:45:23.659 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:45:23.659 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:45:23.659 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:23.659 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:45:23.659 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:45:23.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:45:23.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:45:23.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:45:23.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:45:23.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:45:23.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:45:23.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:45:23.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:45:23.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:23.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:45:23.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:45:23.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:45:23.661 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:45:23.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=bc68bb65-33a0-4802-9e9b-c0869e7c2f4b 2020-03-11T22:45:23.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 bc68bb65-33a0-4802-9e9b-c0869e7c2f4b' 2020-03-11T22:45:23.662 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:45:23.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQADamlebqPwJxAAtwn9KNr5eRyXc22bmJIWGQ== 2020-03-11T22:45:23.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQADamlebqPwJxAAtwn9KNr5eRyXc22bmJIWGQ=="}' 2020-03-11T22:45:23.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new bc68bb65-33a0-4802-9e9b-c0869e7c2f4b -i td/divergent-priors/1/new.json 2020-03-11T22:45:24.225 INFO:tasks.workunit.client.0.smithi145.stdout:1 2020-03-11T22:45:24.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/1/new.json 2020-03-11T22:45:24.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQADamlebqPwJxAAtwn9KNr5eRyXc22bmJIWGQ== --osd-uuid bc68bb65-33a0-4802-9e9b-c0869e7c2f4b 2020-03-11T22:45:24.265 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:45:24.262+0000 7f37500b0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:24.267 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:45:24.264+0000 7f37500b0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:24.269 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:45:24.266+0000 7f37500b0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:24.269 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:45:24.267+0000 7f37500b0ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:45:24.270 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:45:24.267+0000 7f37500b0ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:45:24.270 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:45:24.267+0000 7f37500b0ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:45:24.320 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:45:24.317+0000 7f37500b0ec0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2020-03-11T22:45:27.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/1/keyring 2020-03-11T22:45:27.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:45:27.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:45:27.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:45:27.305 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2020-03-11T22:45:27.759 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2020-03-11T22:45:27.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:45:27.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:27.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:45:27.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:45:27.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:45:27.786 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:45:27.783+0000 7f4ccda2aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:27.787 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:45:27.785+0000 7f4ccda2aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:27.790 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:45:27.787+0000 7f4ccda2aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:28.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:45:28.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:45:28.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:45:28.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:45:28.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:45:28.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:28.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:45:28.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:28.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:45:28.105 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:45:28.305 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:45:28.302+0000 7f4ccda2aec0 -1 Falling back to public interface 2020-03-11T22:45:28.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:45:29.102 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:45:29.099+0000 7f4ccda2aec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:45:29.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:45:29.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:29.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:45:29.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:29.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:45:29.447 INFO:tasks.workunit.client.0.smithi145.stdout:1 2020-03-11T22:45:29.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:45:30.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:45:30.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:30.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 2 2020-03-11T22:45:30.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:30.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:45:30.810 INFO:tasks.workunit.client.0.smithi145.stdout:2 2020-03-11T22:45:31.193 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/43891,v1:127.0.0.1:6811/43891] [v2:127.0.0.1:6812/43891,v1:127.0.0.1:6813/43891] exists,up bc68bb65-33a0-4802-9e9b-c0869e7c2f4b 2020-03-11T22:45:31.193 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 1df352e0-156e-4ee7-a96d-63bceae4b98b 2020-03-11T22:45:31.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:45:31.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:45:31.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:45:31.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2020-03-11T22:45:31.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 2 2020-03-11T22:45:31.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:45:31.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:45:31.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:45:31.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:45:31.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:45:31.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:45:31.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:45:31.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:45:31.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:45:31.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:45:31.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:45:31.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:45:31.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:45:31.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:45:31.199 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:45:31.199 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:45:31.199 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:45:31.199 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:45:31.200 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:31.200 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:45:31.200 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:45:31.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:45:31.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:45:31.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:45:31.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:45:31.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:45:31.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:45:31.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:45:31.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:45:31.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:31.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:45:31.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:45:31.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:45:31.204 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:45:31.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=1df352e0-156e-4ee7-a96d-63bceae4b98b 2020-03-11T22:45:31.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 1df352e0-156e-4ee7-a96d-63bceae4b98b' 2020-03-11T22:45:31.204 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:45:31.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQALamlewxwtDBAAWpu0Etn1+a9KICKJrFVZJg== 2020-03-11T22:45:31.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQALamlewxwtDBAAWpu0Etn1+a9KICKJrFVZJg=="}' 2020-03-11T22:45:31.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 1df352e0-156e-4ee7-a96d-63bceae4b98b -i td/divergent-priors/2/new.json 2020-03-11T22:45:31.667 INFO:tasks.workunit.client.0.smithi145.stdout:2 2020-03-11T22:45:31.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/2/new.json 2020-03-11T22:45:31.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQALamlewxwtDBAAWpu0Etn1+a9KICKJrFVZJg== --osd-uuid 1df352e0-156e-4ee7-a96d-63bceae4b98b 2020-03-11T22:45:31.706 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:45:31.703+0000 7f1de530fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:31.718 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:45:31.715+0000 7f1de530fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:31.719 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:45:31.716+0000 7f1de530fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:31.720 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:45:31.717+0000 7f1de530fec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:45:31.720 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:45:31.717+0000 7f1de530fec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:45:31.720 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:45:31.717+0000 7f1de530fec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:45:31.785 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:45:31.782+0000 7f1de530fec0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2020-03-11T22:45:34.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/2/keyring 2020-03-11T22:45:34.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:45:34.762 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2020-03-11T22:45:34.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:45:34.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:45:35.217 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2020-03-11T22:45:35.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:45:35.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:35.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:45:35.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:45:35.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:45:35.243 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:45:35.240+0000 7fd936a9aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:35.244 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:45:35.242+0000 7fd936a9aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:35.247 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:45:35.244+0000 7fd936a9aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:35.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:45:35.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:45:35.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:45:35.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:45:35.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:45:35.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:35.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:45:35.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:35.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:45:35.595 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:45:35.768 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:45:35.765+0000 7fd936a9aec0 -1 Falling back to public interface 2020-03-11T22:45:35.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:45:36.525 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:45:36.522+0000 7fd936a9aec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:45:36.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:45:36.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:36.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:45:36.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:36.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:45:36.938 INFO:tasks.workunit.client.0.smithi145.stdout:1 2020-03-11T22:45:37.298 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/44274,v1:127.0.0.1:6819/44274] [v2:127.0.0.1:6820/44274,v1:127.0.0.1:6821/44274] exists,up 1df352e0-156e-4ee7-a96d-63bceae4b98b 2020-03-11T22:45:37.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:45:37.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:45:37.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:45:37.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:243: TEST_divergent_ec: ceph osd set noout 2020-03-11T22:45:38.019 INFO:tasks.workunit.client.0.smithi145.stderr:noout is set 2020-03-11T22:45:38.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:244: TEST_divergent_ec: ceph osd set noin 2020-03-11T22:45:38.745 INFO:tasks.workunit.client.0.smithi145.stderr:noin is set 2020-03-11T22:45:38.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:245: TEST_divergent_ec: ceph osd set nodown 2020-03-11T22:45:39.463 INFO:tasks.workunit.client.0.smithi145.stderr:nodown is set 2020-03-11T22:45:39.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:246: TEST_divergent_ec: create_ec_pool test true k=2 m=1 2020-03-11T22:45:39.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: create_ec_pool: local pool_name=test 2020-03-11T22:45:39.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: create_ec_pool: shift 2020-03-11T22:45:39.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: create_ec_pool: local allow_overwrites=true 2020-03-11T22:45:39.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: create_ec_pool: shift 2020-03-11T22:45:39.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2020-03-11T22:45:40.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: create_ec_pool: create_pool test 1 1 erasure myprofile 2020-03-11T22:45:40.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 erasure myprofile 2020-03-11T22:45:40.799 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' already exists 2020-03-11T22:45:40.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:45:41.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: create_ec_pool: '[' true = true ']' 2020-03-11T22:45:41.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: create_ec_pool: ceph osd pool set test allow_ec_overwrites true 2020-03-11T22:45:42.419 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 allow_ec_overwrites to true 2020-03-11T22:45:42.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: create_ec_pool: wait_for_clean 2020-03-11T22:45:42.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:45:42.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:45:42.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:45:42.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:45:42.429 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:45:42.429 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:45:42.430 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:45:42.430 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:45:42.430 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:45:42.430 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:45:42.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:45:42.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:45:42.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:45:42.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:45:42.521 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:45:42.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:45:42.867 INFO:tasks.workunit.client.0.smithi145.stderr:1 2020-03-11T22:45:42.867 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2020-03-11T22:45:42.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:45:42.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:45:42.867 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:45:43.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836486 2020-03-11T22:45:43.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486' 2020-03-11T22:45:43.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:45:43.001 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:45:43.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672966 2020-03-11T22:45:43.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672966' 2020-03-11T22:45:43.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:45:43.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:45:43.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509444 2020-03-11T22:45:43.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672966 2-64424509444' 2020-03-11T22:45:43.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:45:43.260 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836486 2020-03-11T22:45:43.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:45:43.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:45:43.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836486 2020-03-11T22:45:43.262 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:45:43.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836486 2020-03-11T22:45:43.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2020-03-11T22:45:43.263 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836486 2020-03-11T22:45:43.263 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:45:43.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836484 -lt 21474836486 2020-03-11T22:45:43.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:45:44.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:45:44.609 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:45:44.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836484 -lt 21474836486 2020-03-11T22:45:44.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:45:45.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 299 -eq 0 ']' 2020-03-11T22:45:45.961 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:45:46.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836487 -lt 21474836486 2020-03-11T22:45:46.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:45:46.314 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672966 2020-03-11T22:45:46.314 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:45:46.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:45:46.315 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672966 2020-03-11T22:45:46.315 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:45:46.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672966 2020-03-11T22:45:46.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2020-03-11T22:45:46.316 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672966 2020-03-11T22:45:46.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:45:46.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672966 -lt 42949672966 2020-03-11T22:45:46.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:45:46.673 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509444 2020-03-11T22:45:46.673 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:45:46.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:45:46.675 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509444 2020-03-11T22:45:46.675 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:45:46.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509444 2020-03-11T22:45:46.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2020-03-11T22:45:46.676 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:45:46.676 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509444 2020-03-11T22:45:47.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509444 -lt 64424509444 2020-03-11T22:45:47.046 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:45:47.047 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:45:47.047 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:45:47.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:45:47.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:45:47.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:45:47.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:45:47.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:45:47.496 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:45:47.496 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:45:47.496 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:45:47.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:45:47.829 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:45:47.830 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:45:47.830 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:45:48.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:45:48.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:45:48.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:45:48.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: create_ec_pool: return 0 2020-03-11T22:45:48.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:248: TEST_divergent_ec: flush_pg_stats 2020-03-11T22:45:48.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:45:48.270 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:45:48.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:45:48.723 INFO:tasks.workunit.client.0.smithi145.stderr:1 2020-03-11T22:45:48.723 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2020-03-11T22:45:48.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:45:48.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:45:48.724 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:45:48.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836489 2020-03-11T22:45:48.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489' 2020-03-11T22:45:48.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:45:48.853 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:45:48.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672969 2020-03-11T22:45:48.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-42949672969' 2020-03-11T22:45:48.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:45:48.983 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:45:49.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509447 2020-03-11T22:45:49.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-42949672969 2-64424509447' 2020-03-11T22:45:49.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:45:49.112 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836489 2020-03-11T22:45:49.112 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:45:49.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:45:49.113 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836489 2020-03-11T22:45:49.113 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:45:49.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836489 2020-03-11T22:45:49.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2020-03-11T22:45:49.115 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:45:49.115 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836489 2020-03-11T22:45:49.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836489 -lt 21474836489 2020-03-11T22:45:49.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:45:49.458 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672969 2020-03-11T22:45:49.459 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:45:49.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:45:49.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672969 2020-03-11T22:45:49.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:45:49.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672969 2020-03-11T22:45:49.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2020-03-11T22:45:49.461 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:45:49.461 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672969 2020-03-11T22:45:49.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672967 -lt 42949672969 2020-03-11T22:45:49.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:45:50.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:45:50.802 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:45:51.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672970 -lt 42949672969 2020-03-11T22:45:51.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:45:51.182 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509447 2020-03-11T22:45:51.182 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:45:51.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:45:51.183 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509447 2020-03-11T22:45:51.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:45:51.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509447 2020-03-11T22:45:51.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2020-03-11T22:45:51.185 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:45:51.185 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509447 2020-03-11T22:45:51.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509447 -lt 64424509447 2020-03-11T22:45:51.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:249: TEST_divergent_ec: wait_for_clean 2020-03-11T22:45:51.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:45:51.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:45:51.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:45:51.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:45:51.528 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:45:51.528 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:45:51.529 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:45:51.529 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:45:51.529 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:45:51.529 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:45:51.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:45:51.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:45:51.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:45:51.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:45:51.621 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:45:51.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:45:51.992 INFO:tasks.workunit.client.0.smithi145.stderr:1 2020-03-11T22:45:51.992 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2020-03-11T22:45:51.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:45:51.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:45:51.993 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:45:52.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836492 2020-03-11T22:45:52.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492' 2020-03-11T22:45:52.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:45:52.148 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:45:52.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672972 2020-03-11T22:45:52.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492 1-42949672972' 2020-03-11T22:45:52.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:45:52.601 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:45:52.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509449 2020-03-11T22:45:52.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492 1-42949672972 2-64424509449' 2020-03-11T22:45:52.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:45:52.602 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836492 2020-03-11T22:45:52.602 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:45:52.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:45:52.602 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836492 2020-03-11T22:45:52.602 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:45:52.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836492 2020-03-11T22:45:52.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2020-03-11T22:45:52.603 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:45:52.603 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836492 2020-03-11T22:45:52.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836490 -lt 21474836492 2020-03-11T22:45:52.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:45:53.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:45:53.750 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:45:54.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836492 -lt 21474836492 2020-03-11T22:45:54.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:45:54.101 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672972 2020-03-11T22:45:54.102 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:45:54.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:45:54.102 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672972 2020-03-11T22:45:54.103 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:45:54.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672972 2020-03-11T22:45:54.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2020-03-11T22:45:54.104 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672972 2020-03-11T22:45:54.104 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:45:54.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672972 -lt 42949672972 2020-03-11T22:45:54.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:45:54.449 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509449 2020-03-11T22:45:54.449 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:45:54.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:45:54.451 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509449 2020-03-11T22:45:54.451 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:45:54.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509449 2020-03-11T22:45:54.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2020-03-11T22:45:54.452 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:45:54.452 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509449 2020-03-11T22:45:54.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509450 -lt 64424509449 2020-03-11T22:45:54.797 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:45:54.798 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:45:54.798 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:45:55.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:45:55.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:45:55.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:45:55.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:45:55.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:45:55.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:45:55.238 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:45:55.238 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:45:55.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:45:55.573 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:45:55.573 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:45:55.573 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:45:56.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:45:56.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:45:56.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:45:56.014 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: ceph pg dump pgs --format=json 2020-03-11T22:45:56.014 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: jq '.pg_stats[0].up_primary' 2020-03-11T22:45:56.334 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:45:56.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: local divergent=1 2020-03-11T22:45:56.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:253: TEST_divergent_ec: echo 'primary and soon to be divergent is 1' 2020-03-11T22:45:56.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:254: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:45:56.344 INFO:tasks.workunit.client.0.smithi145.stdout:primary and soon to be divergent is 1 2020-03-11T22:45:56.661 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:45:56.661 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:45:40.995752+0000 0'0 26:11 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:45:40.484776+0000 0'0 2020-03-11T22:45:40.484776+0000 0 2020-03-11T22:45:56.661 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:45:56.662 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:45:56.663 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:45:56.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:255: TEST_divergent_ec: local non_divergent= 2020-03-11T22:45:56.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2020-03-11T22:45:56.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 0 = 1 ']' 2020-03-11T22:45:56.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0' 2020-03-11T22:45:56.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2020-03-11T22:45:56.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 1 = 1 ']' 2020-03-11T22:45:56.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:259: TEST_divergent_ec: continue 2020-03-11T22:45:56.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2020-03-11T22:45:56.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 2 = 1 ']' 2020-03-11T22:45:56.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0 2' 2020-03-11T22:45:56.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:264: TEST_divergent_ec: echo 'writing initial objects' 2020-03-11T22:45:56.674 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: seq 1 100 2020-03-11T22:45:56.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:56.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_1 /etc/fstab 2020-03-11T22:45:56.675 INFO:tasks.workunit.client.0.smithi145.stdout:writing initial objects 2020-03-11T22:45:56.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:56.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_2 /etc/fstab 2020-03-11T22:45:56.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:56.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_3 /etc/fstab 2020-03-11T22:45:56.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:56.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_4 /etc/fstab 2020-03-11T22:45:56.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:56.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_5 /etc/fstab 2020-03-11T22:45:56.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:56.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_6 /etc/fstab 2020-03-11T22:45:57.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:57.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_7 /etc/fstab 2020-03-11T22:45:57.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:57.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_8 /etc/fstab 2020-03-11T22:45:57.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:57.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_9 /etc/fstab 2020-03-11T22:45:57.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:57.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_10 /etc/fstab 2020-03-11T22:45:57.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:57.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_11 /etc/fstab 2020-03-11T22:45:57.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:57.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_12 /etc/fstab 2020-03-11T22:45:57.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:57.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_13 /etc/fstab 2020-03-11T22:45:57.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:57.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_14 /etc/fstab 2020-03-11T22:45:57.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:57.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_15 /etc/fstab 2020-03-11T22:45:57.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:57.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_16 /etc/fstab 2020-03-11T22:45:57.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:57.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_17 /etc/fstab 2020-03-11T22:45:57.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:57.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_18 /etc/fstab 2020-03-11T22:45:57.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:57.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_19 /etc/fstab 2020-03-11T22:45:57.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:57.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_20 /etc/fstab 2020-03-11T22:45:57.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:57.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_21 /etc/fstab 2020-03-11T22:45:57.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:57.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_22 /etc/fstab 2020-03-11T22:45:57.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:57.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_23 /etc/fstab 2020-03-11T22:45:58.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:58.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_24 /etc/fstab 2020-03-11T22:45:58.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:58.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_25 /etc/fstab 2020-03-11T22:45:58.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:58.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_26 /etc/fstab 2020-03-11T22:45:58.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:58.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_27 /etc/fstab 2020-03-11T22:45:58.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:58.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_28 /etc/fstab 2020-03-11T22:45:58.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:58.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_29 /etc/fstab 2020-03-11T22:45:58.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:58.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_30 /etc/fstab 2020-03-11T22:45:58.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:58.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_31 /etc/fstab 2020-03-11T22:45:58.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:58.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_32 /etc/fstab 2020-03-11T22:45:58.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:58.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_33 /etc/fstab 2020-03-11T22:45:58.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:58.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_34 /etc/fstab 2020-03-11T22:45:58.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:58.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_35 /etc/fstab 2020-03-11T22:45:58.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:58.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_36 /etc/fstab 2020-03-11T22:45:58.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:58.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_37 /etc/fstab 2020-03-11T22:45:58.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:58.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_38 /etc/fstab 2020-03-11T22:45:58.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:58.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_39 /etc/fstab 2020-03-11T22:45:58.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:58.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_40 /etc/fstab 2020-03-11T22:45:59.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:59.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_41 /etc/fstab 2020-03-11T22:45:59.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:59.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_42 /etc/fstab 2020-03-11T22:45:59.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:59.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_43 /etc/fstab 2020-03-11T22:45:59.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:59.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_44 /etc/fstab 2020-03-11T22:45:59.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:59.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_45 /etc/fstab 2020-03-11T22:45:59.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:59.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_46 /etc/fstab 2020-03-11T22:45:59.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:59.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_47 /etc/fstab 2020-03-11T22:45:59.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:59.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_48 /etc/fstab 2020-03-11T22:45:59.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:59.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_49 /etc/fstab 2020-03-11T22:45:59.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:59.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_50 /etc/fstab 2020-03-11T22:45:59.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:59.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_51 /etc/fstab 2020-03-11T22:45:59.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:59.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_52 /etc/fstab 2020-03-11T22:45:59.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:59.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_53 /etc/fstab 2020-03-11T22:45:59.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:59.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_54 /etc/fstab 2020-03-11T22:45:59.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:59.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_55 /etc/fstab 2020-03-11T22:45:59.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:59.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_56 /etc/fstab 2020-03-11T22:45:59.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:59.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_57 /etc/fstab 2020-03-11T22:45:59.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:45:59.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_58 /etc/fstab 2020-03-11T22:46:00.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:00.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_59 /etc/fstab 2020-03-11T22:46:00.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:00.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_60 /etc/fstab 2020-03-11T22:46:00.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:00.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_61 /etc/fstab 2020-03-11T22:46:00.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:00.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_62 /etc/fstab 2020-03-11T22:46:00.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:00.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_63 /etc/fstab 2020-03-11T22:46:00.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:00.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_64 /etc/fstab 2020-03-11T22:46:00.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:00.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_65 /etc/fstab 2020-03-11T22:46:00.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:00.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_66 /etc/fstab 2020-03-11T22:46:00.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:00.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_67 /etc/fstab 2020-03-11T22:46:00.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:00.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_68 /etc/fstab 2020-03-11T22:46:00.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:00.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_69 /etc/fstab 2020-03-11T22:46:00.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:00.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_70 /etc/fstab 2020-03-11T22:46:00.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:00.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_71 /etc/fstab 2020-03-11T22:46:00.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:00.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_72 /etc/fstab 2020-03-11T22:46:00.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:00.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_73 /etc/fstab 2020-03-11T22:46:00.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:00.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_74 /etc/fstab 2020-03-11T22:46:00.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:00.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_75 /etc/fstab 2020-03-11T22:46:00.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:00.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_76 /etc/fstab 2020-03-11T22:46:01.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:01.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_77 /etc/fstab 2020-03-11T22:46:01.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:01.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_78 /etc/fstab 2020-03-11T22:46:01.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:01.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_79 /etc/fstab 2020-03-11T22:46:01.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:01.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_80 /etc/fstab 2020-03-11T22:46:01.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:01.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_81 /etc/fstab 2020-03-11T22:46:01.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:01.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_82 /etc/fstab 2020-03-11T22:46:01.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:01.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_83 /etc/fstab 2020-03-11T22:46:01.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:01.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_84 /etc/fstab 2020-03-11T22:46:01.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:01.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_85 /etc/fstab 2020-03-11T22:46:01.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:01.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_86 /etc/fstab 2020-03-11T22:46:01.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:01.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_87 /etc/fstab 2020-03-11T22:46:01.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:01.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_88 /etc/fstab 2020-03-11T22:46:01.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:01.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_89 /etc/fstab 2020-03-11T22:46:01.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:01.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_90 /etc/fstab 2020-03-11T22:46:01.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:01.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_91 /etc/fstab 2020-03-11T22:46:01.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:01.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_92 /etc/fstab 2020-03-11T22:46:02.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:02.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_93 /etc/fstab 2020-03-11T22:46:02.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:02.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_94 /etc/fstab 2020-03-11T22:46:02.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:02.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_95 /etc/fstab 2020-03-11T22:46:02.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:02.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_96 /etc/fstab 2020-03-11T22:46:02.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:02.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_97 /etc/fstab 2020-03-11T22:46:02.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:02.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_98 /etc/fstab 2020-03-11T22:46:02.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:02.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2020-03-11T22:46:02.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:02.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2020-03-11T22:46:02.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:46:02.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: wait_for_clean 2020-03-11T22:46:02.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:46:02.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:46:02.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:46:02.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:46:02.479 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:46:02.479 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:46:02.479 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:46:02.479 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:46:02.480 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:46:02.480 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:46:02.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:46:02.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:46:02.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:46:02.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:46:02.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:46:02.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:46:02.877 INFO:tasks.workunit.client.0.smithi145.stderr:1 2020-03-11T22:46:02.877 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2020-03-11T22:46:02.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:46:02.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:02.877 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:46:03.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836496 2020-03-11T22:46:03.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836496' 2020-03-11T22:46:03.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:03.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:46:03.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672976 2020-03-11T22:46:03.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836496 1-42949672976' 2020-03-11T22:46:03.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:03.135 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:46:03.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509454 2020-03-11T22:46:03.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836496 1-42949672976 2-64424509454' 2020-03-11T22:46:03.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:46:03.265 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836496 2020-03-11T22:46:03.265 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:46:03.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:46:03.266 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836496 2020-03-11T22:46:03.266 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:46:03.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836496 2020-03-11T22:46:03.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2020-03-11T22:46:03.267 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:46:03.268 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836496 2020-03-11T22:46:03.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836494 -lt 21474836496 2020-03-11T22:46:03.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:46:04.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:46:04.635 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:46:04.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836494 -lt 21474836496 2020-03-11T22:46:04.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:46:05.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 299 -eq 0 ']' 2020-03-11T22:46:05.986 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:46:06.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836497 -lt 21474836496 2020-03-11T22:46:06.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:46:06.338 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672976 2020-03-11T22:46:06.338 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:46:06.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:46:06.339 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672976 2020-03-11T22:46:06.339 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:46:06.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672976 2020-03-11T22:46:06.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2020-03-11T22:46:06.340 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:46:06.341 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672976 2020-03-11T22:46:06.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672976 -lt 42949672976 2020-03-11T22:46:06.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:46:06.684 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509454 2020-03-11T22:46:06.684 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:46:06.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:46:06.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509454 2020-03-11T22:46:06.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:46:06.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509454 2020-03-11T22:46:06.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2020-03-11T22:46:06.687 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509454 2020-03-11T22:46:06.687 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:46:07.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509454 -lt 64424509454 2020-03-11T22:46:07.044 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:46:07.044 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:07.044 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:07.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:46:07.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:46:07.489 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:46:07.489 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:46:07.489 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:46:07.489 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:46:07.489 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:46:07.489 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:46:07.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:46:07.819 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:46:07.819 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:07.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:08.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:46:08.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:46:08.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:46:08.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: get_pg test existing_1 2020-03-11T22:46:08.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local poolname=test 2020-03-11T22:46:08.262 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local objectname=existing_1 2020-03-11T22:46:08.262 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: ceph --format json osd map test existing_1 2020-03-11T22:46:08.262 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: jq -r .pgid 2020-03-11T22:46:08.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: local pgid=1.0 2020-03-11T22:46:08.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:276: TEST_divergent_ec: echo 'blackholing osds 0 2' 2020-03-11T22:46:08.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:277: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:46:08.606 INFO:tasks.workunit.client.0.smithi145.stdout:blackholing osds 0 2 2020-03-11T22:46:08.926 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:46:08.926 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2020-03-11T22:45:40.995752+0000 28'100 28:111 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:45:40.484776+0000 0'0 2020-03-11T22:45:40.484776+0000 0 2020-03-11T22:46:08.927 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:46:08.927 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:46:08.928 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:46:08.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:46:08.938 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.0 2020-03-11T22:46:08.938 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:46:08.938 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:46:08.938 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:46:08.939 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:08.939 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:46:08.939 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17402/ceph-osd.0.asok 2020-03-11T22:46:08.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2020-03-11T22:46:08.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.17402/ceph-osd.0.asok config set objectstore_blackhole 1 2020-03-11T22:46:09.024 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:46:09.024 INFO:tasks.workunit.client.0.smithi145.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:46:09.024 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:46:09.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:46:09.033 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.2 2020-03-11T22:46:09.033 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:46:09.033 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:46:09.033 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:46:09.034 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:09.034 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:46:09.034 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17402/ceph-osd.2.asok 2020-03-11T22:46:09.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2020-03-11T22:46:09.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.17402/ceph-osd.2.asok config set objectstore_blackhole 1 2020-03-11T22:46:09.119 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:46:09.119 INFO:tasks.workunit.client.0.smithi145.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:46:09.120 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:46:09.126 INFO:tasks.workunit.client.0.smithi145.stdout:writing divergent object 2020-03-11T22:46:09.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:284: TEST_divergent_ec: echo 'writing divergent object' 2020-03-11T22:46:09.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:286: TEST_divergent_ec: sleep 1 2020-03-11T22:46:09.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:285: TEST_divergent_ec: rados -p test put existing_100 /etc/resolv.conf 2020-03-11T22:46:10.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:288: TEST_divergent_ec: rados -p test mksnap snap1 2020-03-11T22:46:10.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:287: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2020-03-11T22:46:10.279 INFO:tasks.workunit.client.0.smithi145.stdout:created pool test snap snap1 2020-03-11T22:46:10.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:290: TEST_divergent_ec: sleep 10 2020-03-11T22:46:10.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: expr 100 - 1 2020-03-11T22:46:10.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2020-03-11T22:46:20.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:291: TEST_divergent_ec: killall -9 rados 2020-03-11T22:46:20.285 INFO:tasks.workunit.client.0.smithi145.stderr:rados: no process found 2020-03-11T22:46:20.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:294: TEST_divergent_ec: echo 'killing all the osds' 2020-03-11T22:46:20.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:295: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:46:20.285 INFO:tasks.workunit.client.0.smithi145.stdout:killing all the osds 2020-03-11T22:46:20.615 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:46:20.615 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 101 0 0 0 0 48783 0 0 14 14 active+clean 2020-03-11T22:45:40.995752+0000 29'104 29:115 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:45:40.484776+0000 0'0 2020-03-11T22:45:40.484776+0000 0 2020-03-11T22:46:20.615 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:46:20.616 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:46:20.617 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:46:20.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:296: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd 2020-03-11T22:46:20.625 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:46:20.626 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:46:20.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:46:20.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:46:20.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:46:20.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:46:20.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2020-03-11T22:46:20.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.0 2020-03-11T22:46:21.265 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already down. 2020-03-11T22:46:21.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2020-03-11T22:46:21.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.1 2020-03-11T22:46:21.766 INFO:tasks.workunit.client.0.smithi145.stderr:osd.1 is already down. 2020-03-11T22:46:21.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2020-03-11T22:46:21.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.2 2020-03-11T22:46:22.348 INFO:tasks.workunit.client.0.smithi145.stderr:osd.2 is already down. 2020-03-11T22:46:22.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:46:22.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.0 2020-03-11T22:46:22.852 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already out. 2020-03-11T22:46:22.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:46:22.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.2 2020-03-11T22:46:23.450 INFO:tasks.workunit.client.0.smithi145.stderr:osd.2 is already out. 2020-03-11T22:46:23.606 INFO:tasks.workunit.client.0.smithi145.stdout:bringing up non_divergent 0 2 2020-03-11T22:46:23.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:307: TEST_divergent_ec: echo 'bringing up non_divergent 0 2' 2020-03-11T22:46:23.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:308: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:46:23.777 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:46:23.777 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 101 0 0 0 0 48783 0 0 14 14 stale+active+clean 2020-03-11T22:45:40.995752+0000 29'104 29:115 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:45:40.484776+0000 0'0 2020-03-11T22:45:40.484776+0000 0 2020-03-11T22:46:23.777 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:46:23.777 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:46:23.778 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:46:23.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:46:23.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 0 2020-03-11T22:46:23.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:46:23.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:46:23.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:46:23.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:46:23.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:46:23.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:46:23.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:46:23.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:46:23.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:46:23.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:46:23.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:46:23.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:46:23.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:46:23.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:46:23.792 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:46:23.792 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:46:23.792 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:46:23.792 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:46:23.792 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:23.793 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:46:23.793 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:46:23.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:46:23.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:46:23.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:46:23.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:46:23.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:46:23.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:46:23.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:23.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:46:23.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:46:23.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:46:23.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:46:23.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:23.794 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-03-11T22:46:23.795 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2020-03-11T22:46:23.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:46:23.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:46:23.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:46:23.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:46:23.815 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:46:23.811+0000 7f4de8884ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:23.816 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:46:23.813+0000 7f4de8884ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:23.818 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:46:23.814+0000 7f4de8884ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:24.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:46:24.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:46:24.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:46:24.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:46:24.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:46:24.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:24.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:46:24.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:24.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:46:24.140 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:46:24.334 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:46:24.330+0000 7f4de8884ec0 -1 Falling back to public interface 2020-03-11T22:46:24.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:46:25.175 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:46:25.171+0000 7f4de8884ec0 -1 osd.0 28 log_to_monitors {default=true} 2020-03-11T22:46:25.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:46:25.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:25.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:46:25.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:25.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:46:25.489 INFO:tasks.workunit.client.0.smithi145.stdout:1 2020-03-11T22:46:25.837 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up out weight 0 up_from 35 up_thru 0 down_at 30 last_clean_interval [5,28) [v2:127.0.0.1:6802/48751,v1:127.0.0.1:6803/48751] [v2:127.0.0.1:6804/48751,v1:127.0.0.1:6805/48751] exists,up 04216e1d-6638-4e33-8cad-1ea8a9c30138 2020-03-11T22:46:25.837 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2020-03-11T22:46:25.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:46:25.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:46:25.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:46:25.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:46:25.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 2 2020-03-11T22:46:25.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:46:25.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:46:25.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:46:25.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:46:25.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:46:25.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:46:25.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:46:25.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:46:25.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:46:25.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:46:25.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:46:25.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:46:25.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:46:25.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:46:25.840 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:46:25.840 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:46:25.840 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:46:25.840 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:46:25.841 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:25.841 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:46:25.841 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:46:25.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:46:25.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:46:25.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:46:25.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:46:25.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:46:25.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:46:25.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:25.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:46:25.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:46:25.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:46:25.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:46:25.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:25.843 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-03-11T22:46:25.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:46:25.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:46:25.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:46:25.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:46:25.862 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:46:25.859+0000 7f93380d0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:25.864 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:46:25.860+0000 7f93380d0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:25.866 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:46:25.862+0000 7f93380d0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:26.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:46:26.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:46:26.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:46:26.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:46:26.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:46:26.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:26.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:46:26.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:46:26.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:26.210 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:46:26.381 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:46:26.377+0000 7f93380d0ec0 -1 Falling back to public interface 2020-03-11T22:46:26.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:46:27.323 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:46:27.319+0000 7f93380d0ec0 -1 osd.2 28 log_to_monitors {default=true} 2020-03-11T22:46:27.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:46:27.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:27.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:46:27.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:27.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:46:27.554 INFO:tasks.workunit.client.0.smithi145.stdout:1 2020-03-11T22:46:27.907 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up out weight 0 up_from 36 up_thru 0 down_at 32 last_clean_interval [15,28) [v2:127.0.0.1:6810/48945,v1:127.0.0.1:6811/48945] [v2:127.0.0.1:6812/48945,v1:127.0.0.1:6813/48945] exists,up 1df352e0-156e-4ee7-a96d-63bceae4b98b 2020-03-11T22:46:27.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:46:27.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:46:27.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:46:27.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:46:27.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.0 2020-03-11T22:46:28.400 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already in. 2020-03-11T22:46:28.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:46:28.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.2 2020-03-11T22:46:29.160 INFO:tasks.workunit.client.0.smithi145.stderr:osd.2 is already in. 2020-03-11T22:46:29.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:318: TEST_divergent_ec: sleep 5 2020-03-11T22:46:34.170 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: expr 2 + 2 2020-03-11T22:46:34.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: objname=existing_4 2020-03-11T22:46:34.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:323: TEST_divergent_ec: echo 'writing non-divergent object existing_4' 2020-03-11T22:46:34.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:324: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:46:34.171 INFO:tasks.workunit.client.0.smithi145.stdout:writing non-divergent object existing_4 2020-03-11T22:46:34.502 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:46:34.502 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded+wait 2020-03-11T22:46:29.224409+0000 28'100 41:129 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-03-11T22:45:40.484776+0000 0'0 2020-03-11T22:45:40.484776+0000 0 2020-03-11T22:46:34.502 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:46:34.502 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:46:34.503 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:46:34.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:325: TEST_divergent_ec: rados -p test put existing_4 /etc/resolv.conf 2020-03-11T22:46:44.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:46:44.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: wait_for_clean 2020-03-11T22:46:44.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:46:44.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:46:44.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:46:44.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:46:44.879 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:46:44.880 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:46:44.880 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:46:44.880 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:46:44.880 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:46:44.880 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:46:44.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:46:44.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:46:44.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:46:44.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:46:44.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:46:45.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:46:45.313 INFO:tasks.workunit.client.0.smithi145.stderr:1 2020-03-11T22:46:45.314 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2020-03-11T22:46:45.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:46:45.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:45.314 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:46:45.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=150323855367 2020-03-11T22:46:45.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-150323855367' 2020-03-11T22:46:45.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:45.444 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:46:45.534 INFO:tasks.workunit.client.0.smithi145.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:46:45.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:46:45.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-150323855367 1-' 2020-03-11T22:46:45.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:45.543 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:46:45.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=154618822662 2020-03-11T22:46:45.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-150323855367 1- 2-154618822662' 2020-03-11T22:46:45.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:46:45.671 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-150323855367 2020-03-11T22:46:45.671 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:46:45.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:46:45.672 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-150323855367 2020-03-11T22:46:45.672 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:46:45.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=150323855367 2020-03-11T22:46:45.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 150323855367' 2020-03-11T22:46:45.674 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 150323855367 2020-03-11T22:46:45.674 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:46:46.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855364 -lt 150323855367 2020-03-11T22:46:46.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:46:47.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:46:47.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:46:47.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855367 -lt 150323855367 2020-03-11T22:46:47.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:46:47.416 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:46:47.416 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:46:47.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:46:47.417 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:46:47.417 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:46:47.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:46:47.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:46:47.418 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:46:47.419 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 2020-03-11T22:46:47.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672980 -lt 2020-03-11T22:46:47.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672980: unary operator expected 2020-03-11T22:46:47.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:46:47.761 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-154618822662 2020-03-11T22:46:47.761 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:46:47.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:46:47.762 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-154618822662 2020-03-11T22:46:47.762 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:46:47.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=154618822662 2020-03-11T22:46:47.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 154618822662' 2020-03-11T22:46:47.763 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:46:47.763 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 154618822662 2020-03-11T22:46:48.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 154618822662 -lt 154618822662 2020-03-11T22:46:48.125 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:46:48.125 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:48.125 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:48.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:46:48.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:46:48.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:46:48.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:46:48.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:46:48.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:46:48.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:46:48.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:46:48.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:46:48.910 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:46:48.910 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:48.911 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:49.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:46:49.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:46:49.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:46:49.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:46:49.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:46:49.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:46:49.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:46:49.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:46:49.453 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:46:49.454 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:46:49.454 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:46:49.454 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:46:49.454 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:46:49.454 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:46:49.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:46:49.786 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:46:49.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:49.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:50.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:46:50.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:46:50.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:46:50.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:46:50.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:46:50.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:46:50.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:46:50.230 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:46:50.230 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:46:50.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:46:50.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:46:50.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:46:50.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:46:50.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:46:50.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:46:50.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:46:50.880 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:46:50.880 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:46:50.881 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:46:50.881 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:46:50.881 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:46:50.881 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:46:51.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:46:51.213 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:46:51.214 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:51.214 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:51.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:46:51.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:46:51.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:46:51.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:46:51.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:46:51.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:46:51.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:46:51.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:46:51.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:46:52.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:46:52.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:46:52.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:46:52.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:46:52.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:46:52.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:46:52.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:46:52.518 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:46:52.518 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:46:52.518 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:46:52.518 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:46:52.519 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:46:52.519 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:46:52.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:46:52.850 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:46:52.850 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:52.850 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:53.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:46:53.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:46:53.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:46:53.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:46:53.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:46:53.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:46:53.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:46:53.294 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:46:53.294 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:46:53.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:46:53.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:46:53.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:46:53.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:46:53.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:46:54.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:46:54.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:46:54.538 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:46:54.539 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:46:54.539 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:46:54.539 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:46:54.539 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:46:54.539 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:46:54.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:46:54.869 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:46:54.870 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:54.870 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:55.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:46:55.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:46:55.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:46:55.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:46:55.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:46:55.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:46:55.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:46:55.321 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:46:55.321 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:46:55.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:46:55.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:46:55.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:46:55.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:46:55.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:46:57.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:46:57.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:46:57.363 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:46:57.363 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:46:57.363 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:46:57.364 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:46:57.364 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:46:57.364 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:46:57.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:46:57.703 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:46:57.703 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:57.704 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:58.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:46:58.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:46:58.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:46:58.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:46:58.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:46:58.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:46:58.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:46:58.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:46:58.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:46:58.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:46:58.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:46:58.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:46:58.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:46:58.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:47:01.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:01.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:01.784 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:01.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:01.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:01.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:01.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:01.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:02.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:02.124 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:02.125 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:02.125 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:02.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:47:02.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:02.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:02.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:02.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:02.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:02.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:02.566 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:02.566 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:03.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:47:03.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:47:03.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:47:03.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:03.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:47:09.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:09.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:09.420 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:09.421 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:09.421 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:09.421 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:09.421 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:09.421 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:09.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:09.762 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:09.763 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:09.763 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:10.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:47:10.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:10.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:10.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:10.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:10.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:10.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:10.207 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:10.207 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:10.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:47:10.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:47:10.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:47:10.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:10.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:47:17.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:17.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:17.950 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:17.950 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:17.950 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:17.950 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:17.951 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:17.951 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:18.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:18.292 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:18.292 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:18.292 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:18.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:47:18.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:18.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:18.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:18.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:18.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:18.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:18.737 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:18.737 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:19.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:47:19.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:47:19.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:47:19.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:47:19.636 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:47:19.636 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_fingerprint": "08e61c9d-a9f7-449a-83c3-d393604848c8", 2020-03-11T22:47:19.636 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:47:19.636 INFO:tasks.workunit.client.0.smithi145.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:47:19.636 INFO:tasks.workunit.client.0.smithi145.stdout: "timestamp": "2020-03-11T22:47:19.632327+0000", 2020-03-11T22:47:19.636 INFO:tasks.workunit.client.0.smithi145.stdout: "tag": "", 2020-03-11T22:47:19.637 INFO:tasks.workunit.client.0.smithi145.stdout: "health": { 2020-03-11T22:47:19.637 INFO:tasks.workunit.client.0.smithi145.stdout: "status": "HEALTH_WARN", 2020-03-11T22:47:19.637 INFO:tasks.workunit.client.0.smithi145.stdout: "checks": { 2020-03-11T22:47:19.637 INFO:tasks.workunit.client.0.smithi145.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:47:19.637 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:47:19.637 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2020-03-11T22:47:19.637 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:47:19.638 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 17 2020-03-11T22:47:19.638 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.638 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [], 2020-03-11T22:47:19.638 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2020-03-11T22:47:19.638 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.638 INFO:tasks.workunit.client.0.smithi145.stdout: "OSD_DOWN": { 2020-03-11T22:47:19.638 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:47:19.638 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2020-03-11T22:47:19.639 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "1 osds down", 2020-03-11T22:47:19.639 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 1 2020-03-11T22:47:19.639 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.639 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2020-03-11T22:47:19.639 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.639 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "osd.1 (root=default,host=smithi145) is down" 2020-03-11T22:47:19.640 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:19.640 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:47:19.640 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2020-03-11T22:47:19.640 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.640 INFO:tasks.workunit.client.0.smithi145.stdout: "PG_DEGRADED": { 2020-03-11T22:47:19.640 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:47:19.640 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2020-03-11T22:47:19.640 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2020-03-11T22:47:19.641 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 1 2020-03-11T22:47:19.641 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.641 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2020-03-11T22:47:19.641 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.641 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [2147483647,0,2]" 2020-03-11T22:47:19.641 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:19.641 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:47:19.642 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2020-03-11T22:47:19.642 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.642 INFO:tasks.workunit.client.0.smithi145.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:47:19.642 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:47:19.642 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2020-03-11T22:47:19.643 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:47:19.643 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 1 2020-03-11T22:47:19.643 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.643 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2020-03-11T22:47:19.643 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.643 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:47:19.643 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.644 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.644 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:47:19.644 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:19.644 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:47:19.644 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2020-03-11T22:47:19.644 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:19.644 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.645 INFO:tasks.workunit.client.0.smithi145.stdout: "mutes": [] 2020-03-11T22:47:19.645 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.645 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap_first_committed": 1, 2020-03-11T22:47:19.645 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap_last_committed": 1, 2020-03-11T22:47:19.645 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap": { 2020-03-11T22:47:19.645 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 1, 2020-03-11T22:47:19.646 INFO:tasks.workunit.client.0.smithi145.stdout: "fsid": "e665ac71-d8d3-412f-b706-8f1d72482f80", 2020-03-11T22:47:19.646 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2020-03-11T22:45:16.173529Z", 2020-03-11T22:47:19.646 INFO:tasks.workunit.client.0.smithi145.stdout: "created": "2020-03-11T22:45:16.173529Z", 2020-03-11T22:47:19.646 INFO:tasks.workunit.client.0.smithi145.stdout: "min_mon_release": 15, 2020-03-11T22:47:19.646 INFO:tasks.workunit.client.0.smithi145.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:47:19.646 INFO:tasks.workunit.client.0.smithi145.stdout: "features": { 2020-03-11T22:47:19.646 INFO:tasks.workunit.client.0.smithi145.stdout: "persistent": [ 2020-03-11T22:47:19.647 INFO:tasks.workunit.client.0.smithi145.stdout: "kraken", 2020-03-11T22:47:19.647 INFO:tasks.workunit.client.0.smithi145.stdout: "luminous", 2020-03-11T22:47:19.647 INFO:tasks.workunit.client.0.smithi145.stdout: "mimic", 2020-03-11T22:47:19.647 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap-prune", 2020-03-11T22:47:19.647 INFO:tasks.workunit.client.0.smithi145.stdout: "nautilus", 2020-03-11T22:47:19.647 INFO:tasks.workunit.client.0.smithi145.stdout: "octopus" 2020-03-11T22:47:19.647 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:47:19.648 INFO:tasks.workunit.client.0.smithi145.stdout: "optional": [] 2020-03-11T22:47:19.648 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.648 INFO:tasks.workunit.client.0.smithi145.stdout: "mons": [ 2020-03-11T22:47:19.648 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.648 INFO:tasks.workunit.client.0.smithi145.stdout: "rank": 0, 2020-03-11T22:47:19.648 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "a", 2020-03-11T22:47:19.648 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2020-03-11T22:47:19.649 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:47:19.649 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.649 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:47:19.649 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:47:19.649 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 0 2020-03-11T22:47:19.649 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:19.649 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:19.650 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.650 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:47:19.650 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:47:19.650 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 0, 2020-03-11T22:47:19.650 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 0 2020-03-11T22:47:19.650 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:19.650 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:19.651 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.651 INFO:tasks.workunit.client.0.smithi145.stdout: "quorum": [ 2020-03-11T22:47:19.651 INFO:tasks.workunit.client.0.smithi145.stdout: 0 2020-03-11T22:47:19.651 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:47:19.651 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap": { 2020-03-11T22:47:19.651 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 41, 2020-03-11T22:47:19.652 INFO:tasks.workunit.client.0.smithi145.stdout: "fsid": "e665ac71-d8d3-412f-b706-8f1d72482f80", 2020-03-11T22:47:19.652 INFO:tasks.workunit.client.0.smithi145.stdout: "created": "2020-03-11T22:45:16.868261+0000", 2020-03-11T22:47:19.652 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2020-03-11T22:46:29.055200+0000", 2020-03-11T22:47:19.652 INFO:tasks.workunit.client.0.smithi145.stdout: "last_up_change": "2020-03-11T22:46:27.393654+0000", 2020-03-11T22:47:19.652 INFO:tasks.workunit.client.0.smithi145.stdout: "last_in_change": "2020-03-11T22:46:28.804829+0000", 2020-03-11T22:47:19.652 INFO:tasks.workunit.client.0.smithi145.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:47:19.652 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_num": 5800384, 2020-03-11T22:47:19.653 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_set": [ 2020-03-11T22:47:19.653 INFO:tasks.workunit.client.0.smithi145.stdout: "nodown", 2020-03-11T22:47:19.653 INFO:tasks.workunit.client.0.smithi145.stdout: "noin", 2020-03-11T22:47:19.653 INFO:tasks.workunit.client.0.smithi145.stdout: "noout", 2020-03-11T22:47:19.653 INFO:tasks.workunit.client.0.smithi145.stdout: "pglog_hardlimit", 2020-03-11T22:47:19.654 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snapdirs", 2020-03-11T22:47:19.654 INFO:tasks.workunit.client.0.smithi145.stdout: "recovery_deletes", 2020-03-11T22:47:19.654 INFO:tasks.workunit.client.0.smithi145.stdout: "sortbitwise" 2020-03-11T22:47:19.654 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:47:19.654 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_version": 8, 2020-03-11T22:47:19.654 INFO:tasks.workunit.client.0.smithi145.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:47:19.654 INFO:tasks.workunit.client.0.smithi145.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:47:19.655 INFO:tasks.workunit.client.0.smithi145.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:47:19.655 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_snapshot": "", 2020-03-11T22:47:19.655 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_max": 1, 2020-03-11T22:47:19.655 INFO:tasks.workunit.client.0.smithi145.stdout: "max_osd": 3, 2020-03-11T22:47:19.655 INFO:tasks.workunit.client.0.smithi145.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:47:19.655 INFO:tasks.workunit.client.0.smithi145.stdout: "min_compat_client": "jewel", 2020-03-11T22:47:19.656 INFO:tasks.workunit.client.0.smithi145.stdout: "require_osd_release": "octopus", 2020-03-11T22:47:19.656 INFO:tasks.workunit.client.0.smithi145.stdout: "pools": [ 2020-03-11T22:47:19.656 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.656 INFO:tasks.workunit.client.0.smithi145.stdout: "pool": 1, 2020-03-11T22:47:19.656 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_name": "test", 2020-03-11T22:47:19.657 INFO:tasks.workunit.client.0.smithi145.stdout: "create_time": "2020-03-11T22:45:40.413891+0000", 2020-03-11T22:47:19.657 INFO:tasks.workunit.client.0.smithi145.stdout: "flags": 16389, 2020-03-11T22:47:19.657 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2020-03-11T22:47:19.657 INFO:tasks.workunit.client.0.smithi145.stdout: "type": 3, 2020-03-11T22:47:19.657 INFO:tasks.workunit.client.0.smithi145.stdout: "size": 3, 2020-03-11T22:47:19.657 INFO:tasks.workunit.client.0.smithi145.stdout: "min_size": 2, 2020-03-11T22:47:19.657 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_rule": 1, 2020-03-11T22:47:19.658 INFO:tasks.workunit.client.0.smithi145.stdout: "object_hash": 2, 2020-03-11T22:47:19.658 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_autoscale_mode": "off", 2020-03-11T22:47:19.658 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num": 1, 2020-03-11T22:47:19.658 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_placement_num": 1, 2020-03-11T22:47:19.658 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_placement_num_target": 1, 2020-03-11T22:47:19.658 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num_target": 1, 2020-03-11T22:47:19.658 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num_pending": 1, 2020-03-11T22:47:19.659 INFO:tasks.workunit.client.0.smithi145.stdout: "last_pg_merge_meta": { 2020-03-11T22:47:19.659 INFO:tasks.workunit.client.0.smithi145.stdout: "source_pgid": "0.0", 2020-03-11T22:47:19.659 INFO:tasks.workunit.client.0.smithi145.stdout: "ready_epoch": 0, 2020-03-11T22:47:19.659 INFO:tasks.workunit.client.0.smithi145.stdout: "last_epoch_started": 0, 2020-03-11T22:47:19.659 INFO:tasks.workunit.client.0.smithi145.stdout: "last_epoch_clean": 0, 2020-03-11T22:47:19.659 INFO:tasks.workunit.client.0.smithi145.stdout: "source_version": "0'0", 2020-03-11T22:47:19.659 INFO:tasks.workunit.client.0.smithi145.stdout: "target_version": "0'0" 2020-03-11T22:47:19.660 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.660 INFO:tasks.workunit.client.0.smithi145.stdout: "last_change": "29", 2020-03-11T22:47:19.660 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend": "0", 2020-03-11T22:47:19.660 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:47:19.660 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend_preluminous": "0", 2020-03-11T22:47:19.660 INFO:tasks.workunit.client.0.smithi145.stdout: "auid": 0, 2020-03-11T22:47:19.661 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_mode": "pool", 2020-03-11T22:47:19.661 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_seq": 1, 2020-03-11T22:47:19.661 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_epoch": 29, 2020-03-11T22:47:19.661 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_snaps": [ 2020-03-11T22:47:19.661 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.661 INFO:tasks.workunit.client.0.smithi145.stdout: "snapid": 1, 2020-03-11T22:47:19.662 INFO:tasks.workunit.client.0.smithi145.stdout: "stamp": "2020-03-11T22:46:10.156100+0000", 2020-03-11T22:47:19.662 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "snap1" 2020-03-11T22:47:19.662 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:19.662 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:47:19.662 INFO:tasks.workunit.client.0.smithi145.stdout: "removed_snaps": "[]", 2020-03-11T22:47:19.662 INFO:tasks.workunit.client.0.smithi145.stdout: "quota_max_bytes": 0, 2020-03-11T22:47:19.662 INFO:tasks.workunit.client.0.smithi145.stdout: "quota_max_objects": 0, 2020-03-11T22:47:19.663 INFO:tasks.workunit.client.0.smithi145.stdout: "tiers": [], 2020-03-11T22:47:19.663 INFO:tasks.workunit.client.0.smithi145.stdout: "tier_of": -1, 2020-03-11T22:47:19.663 INFO:tasks.workunit.client.0.smithi145.stdout: "read_tier": -1, 2020-03-11T22:47:19.663 INFO:tasks.workunit.client.0.smithi145.stdout: "write_tier": -1, 2020-03-11T22:47:19.663 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_mode": "none", 2020-03-11T22:47:19.663 INFO:tasks.workunit.client.0.smithi145.stdout: "target_max_bytes": 0, 2020-03-11T22:47:19.664 INFO:tasks.workunit.client.0.smithi145.stdout: "target_max_objects": 0, 2020-03-11T22:47:19.664 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:47:19.664 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:47:19.664 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:47:19.664 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_min_flush_age": 0, 2020-03-11T22:47:19.664 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_min_evict_age": 0, 2020-03-11T22:47:19.664 INFO:tasks.workunit.client.0.smithi145.stdout: "erasure_code_profile": "myprofile", 2020-03-11T22:47:19.665 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_params": { 2020-03-11T22:47:19.665 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "none" 2020-03-11T22:47:19.665 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.665 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_period": 0, 2020-03-11T22:47:19.665 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_count": 0, 2020-03-11T22:47:19.665 INFO:tasks.workunit.client.0.smithi145.stdout: "use_gmt_hitset": true, 2020-03-11T22:47:19.665 INFO:tasks.workunit.client.0.smithi145.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:47:19.666 INFO:tasks.workunit.client.0.smithi145.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:47:19.666 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:47:19.666 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:47:19.666 INFO:tasks.workunit.client.0.smithi145.stdout: "grade_table": [], 2020-03-11T22:47:19.666 INFO:tasks.workunit.client.0.smithi145.stdout: "stripe_width": 8192, 2020-03-11T22:47:19.666 INFO:tasks.workunit.client.0.smithi145.stdout: "expected_num_objects": 0, 2020-03-11T22:47:19.666 INFO:tasks.workunit.client.0.smithi145.stdout: "fast_read": false, 2020-03-11T22:47:19.667 INFO:tasks.workunit.client.0.smithi145.stdout: "options": {}, 2020-03-11T22:47:19.667 INFO:tasks.workunit.client.0.smithi145.stdout: "application_metadata": {} 2020-03-11T22:47:19.667 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:19.667 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:47:19.667 INFO:tasks.workunit.client.0.smithi145.stdout: "osds": [ 2020-03-11T22:47:19.667 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.667 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2020-03-11T22:47:19.667 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "04216e1d-6638-4e33-8cad-1ea8a9c30138", 2020-03-11T22:47:19.668 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 1, 2020-03-11T22:47:19.668 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2020-03-11T22:47:19.668 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2020-03-11T22:47:19.668 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2020-03-11T22:47:19.668 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 5, 2020-03-11T22:47:19.668 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 28, 2020-03-11T22:47:19.668 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 35, 2020-03-11T22:47:19.668 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 40, 2020-03-11T22:47:19.669 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 30, 2020-03-11T22:47:19.669 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2020-03-11T22:47:19.669 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2020-03-11T22:47:19.669 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:47:19.669 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.669 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:47:19.669 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:47:19.670 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 48751 2020-03-11T22:47:19.670 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.670 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.670 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:47:19.670 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:47:19.670 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 48751 2020-03-11T22:47:19.671 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:19.671 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:19.671 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.671 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2020-03-11T22:47:19.671 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:47:19.671 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.671 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:47:19.672 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:47:19.672 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 48751 2020-03-11T22:47:19.672 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.672 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.672 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:47:19.673 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:47:19.673 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 48751 2020-03-11T22:47:19.673 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:19.673 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:19.673 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.673 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2020-03-11T22:47:19.673 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:47:19.673 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.674 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:47:19.674 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:47:19.674 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 48751 2020-03-11T22:47:19.674 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.674 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.674 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:47:19.675 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:47:19.675 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 48751 2020-03-11T22:47:19.675 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:19.675 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:19.675 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.675 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2020-03-11T22:47:19.675 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:47:19.675 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.676 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:47:19.676 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:47:19.676 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 48751 2020-03-11T22:47:19.676 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.676 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.676 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:47:19.676 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:47:19.677 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 48751 2020-03-11T22:47:19.677 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:19.677 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:19.677 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.677 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6803/48751", 2020-03-11T22:47:19.677 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6805/48751", 2020-03-11T22:47:19.678 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6809/48751", 2020-03-11T22:47:19.678 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6807/48751", 2020-03-11T22:47:19.678 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2020-03-11T22:47:19.678 INFO:tasks.workunit.client.0.smithi145.stdout: "exists", 2020-03-11T22:47:19.678 INFO:tasks.workunit.client.0.smithi145.stdout: "up" 2020-03-11T22:47:19.678 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:19.679 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.679 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.679 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2020-03-11T22:47:19.679 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "bc68bb65-33a0-4802-9e9b-c0869e7c2f4b", 2020-03-11T22:47:19.679 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 0, 2020-03-11T22:47:19.679 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2020-03-11T22:47:19.679 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2020-03-11T22:47:19.680 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2020-03-11T22:47:19.680 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 0, 2020-03-11T22:47:19.680 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 0, 2020-03-11T22:47:19.680 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 10, 2020-03-11T22:47:19.680 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 24, 2020-03-11T22:47:19.680 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 31, 2020-03-11T22:47:19.680 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2020-03-11T22:47:19.681 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2020-03-11T22:47:19.681 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:47:19.681 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.681 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:47:19.681 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:47:19.681 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 43891 2020-03-11T22:47:19.682 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.682 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.682 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:47:19.682 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:47:19.682 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 43891 2020-03-11T22:47:19.682 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:19.682 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:19.682 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.683 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2020-03-11T22:47:19.683 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:47:19.683 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.683 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:47:19.683 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:47:19.683 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 43891 2020-03-11T22:47:19.684 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.684 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.684 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:47:19.684 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:47:19.684 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 43891 2020-03-11T22:47:19.684 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:19.684 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:19.685 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.685 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2020-03-11T22:47:19.685 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:47:19.685 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.685 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:47:19.685 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:47:19.685 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 43891 2020-03-11T22:47:19.686 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.686 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.686 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:47:19.686 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:47:19.686 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 43891 2020-03-11T22:47:19.686 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:19.687 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:19.687 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.687 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2020-03-11T22:47:19.687 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:47:19.687 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.687 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:47:19.687 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:47:19.688 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 43891 2020-03-11T22:47:19.688 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.688 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.688 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:47:19.688 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:47:19.688 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 43891 2020-03-11T22:47:19.688 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:19.689 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:19.689 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.689 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6811/43891", 2020-03-11T22:47:19.689 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6813/43891", 2020-03-11T22:47:19.689 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6817/43891", 2020-03-11T22:47:19.689 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6815/43891", 2020-03-11T22:47:19.690 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2020-03-11T22:47:19.690 INFO:tasks.workunit.client.0.smithi145.stdout: "exists" 2020-03-11T22:47:19.690 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:19.690 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.690 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.690 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2020-03-11T22:47:19.690 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "1df352e0-156e-4ee7-a96d-63bceae4b98b", 2020-03-11T22:47:19.690 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 1, 2020-03-11T22:47:19.691 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2020-03-11T22:47:19.691 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2020-03-11T22:47:19.691 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2020-03-11T22:47:19.691 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 15, 2020-03-11T22:47:19.691 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 28, 2020-03-11T22:47:19.691 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 36, 2020-03-11T22:47:19.691 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 0, 2020-03-11T22:47:19.692 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 32, 2020-03-11T22:47:19.692 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2020-03-11T22:47:19.692 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2020-03-11T22:47:19.692 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:47:19.692 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.692 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:47:19.692 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:47:19.693 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 48945 2020-03-11T22:47:19.693 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.693 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.693 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:47:19.693 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:47:19.693 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 48945 2020-03-11T22:47:19.693 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:19.694 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:19.694 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.694 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2020-03-11T22:47:19.694 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:47:19.695 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.695 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:47:19.695 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:47:19.695 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 48945 2020-03-11T22:47:19.695 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.695 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.695 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:47:19.695 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:47:19.696 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 48945 2020-03-11T22:47:19.696 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:19.696 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:19.696 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.696 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2020-03-11T22:47:19.696 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:47:19.696 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.696 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:47:19.697 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:47:19.697 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 48945 2020-03-11T22:47:19.697 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.697 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.697 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:47:19.697 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:47:19.697 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 48945 2020-03-11T22:47:19.698 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:19.698 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:19.698 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.698 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2020-03-11T22:47:19.698 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:47:19.698 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.699 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:47:19.699 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:47:19.699 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 48945 2020-03-11T22:47:19.699 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.699 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.699 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:47:19.699 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:47:19.700 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 48945 2020-03-11T22:47:19.700 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:19.700 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:19.700 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.700 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6811/48945", 2020-03-11T22:47:19.700 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6813/48945", 2020-03-11T22:47:19.701 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6817/48945", 2020-03-11T22:47:19.701 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6815/48945", 2020-03-11T22:47:19.701 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2020-03-11T22:47:19.701 INFO:tasks.workunit.client.0.smithi145.stdout: "exists", 2020-03-11T22:47:19.701 INFO:tasks.workunit.client.0.smithi145.stdout: "up" 2020-03-11T22:47:19.701 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:19.701 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:19.702 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:47:19.702 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_xinfo": [ 2020-03-11T22:47:19.702 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.702 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2020-03-11T22:47:19.702 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "2020-03-11T22:46:21.027039+0000", 2020-03-11T22:47:19.702 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2020-03-11T22:47:19.702 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2020-03-11T22:47:19.703 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138292836696063, 2020-03-11T22:47:19.703 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2020-03-11T22:47:19.704 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2020-03-11T22:45:23.858878+0000", 2020-03-11T22:47:19.704 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2020-03-11T22:47:19.704 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.704 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.704 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2020-03-11T22:47:19.704 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "2020-03-11T22:46:21.551717+0000", 2020-03-11T22:47:19.705 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2020-03-11T22:47:19.705 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2020-03-11T22:47:19.705 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138292836696063, 2020-03-11T22:47:19.705 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2020-03-11T22:47:19.705 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2020-03-11T22:45:30.069381+0000", 2020-03-11T22:47:19.705 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2020-03-11T22:47:19.705 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.706 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.706 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2020-03-11T22:47:19.706 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "2020-03-11T22:46:22.135997+0000", 2020-03-11T22:47:19.706 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2020-03-11T22:47:19.706 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2020-03-11T22:47:19.706 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138292836696063, 2020-03-11T22:47:19.706 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2020-03-11T22:47:19.707 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2020-03-11T22:45:37.532782+0000", 2020-03-11T22:47:19.707 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2020-03-11T22:47:19.707 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:19.707 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:47:19.707 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_upmap": [], 2020-03-11T22:47:19.707 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_upmap_items": [], 2020-03-11T22:47:19.707 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_temp": [], 2020-03-11T22:47:19.708 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_temp": [], 2020-03-11T22:47:19.708 INFO:tasks.workunit.client.0.smithi145.stdout: "blacklist": {}, 2020-03-11T22:47:19.708 INFO:tasks.workunit.client.0.smithi145.stdout: "erasure_code_profiles": { 2020-03-11T22:47:19.708 INFO:tasks.workunit.client.0.smithi145.stdout: "default": { 2020-03-11T22:47:19.708 INFO:tasks.workunit.client.0.smithi145.stdout: "k": "2", 2020-03-11T22:47:19.708 INFO:tasks.workunit.client.0.smithi145.stdout: "m": "2", 2020-03-11T22:47:19.709 INFO:tasks.workunit.client.0.smithi145.stdout: "plugin": "jerasure", 2020-03-11T22:47:19.709 INFO:tasks.workunit.client.0.smithi145.stdout: "technique": "reed_sol_van" 2020-03-11T22:47:19.709 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.709 INFO:tasks.workunit.client.0.smithi145.stdout: "myprofile": { 2020-03-11T22:47:19.709 INFO:tasks.workunit.client.0.smithi145.stdout: "crush-device-class": "", 2020-03-11T22:47:19.709 INFO:tasks.workunit.client.0.smithi145.stdout: "crush-failure-domain": "osd", 2020-03-11T22:47:19.709 INFO:tasks.workunit.client.0.smithi145.stdout: "crush-root": "default", 2020-03-11T22:47:19.709 INFO:tasks.workunit.client.0.smithi145.stdout: "jerasure-per-chunk-alignment": "false", 2020-03-11T22:47:19.710 INFO:tasks.workunit.client.0.smithi145.stdout: "k": "2", 2020-03-11T22:47:19.710 INFO:tasks.workunit.client.0.smithi145.stdout: "m": "1", 2020-03-11T22:47:19.710 INFO:tasks.workunit.client.0.smithi145.stdout: "plugin": "jerasure", 2020-03-11T22:47:19.710 INFO:tasks.workunit.client.0.smithi145.stdout: "technique": "reed_sol_van", 2020-03-11T22:47:19.710 INFO:tasks.workunit.client.0.smithi145.stdout: "w": "8" 2020-03-11T22:47:19.710 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:19.710 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.711 INFO:tasks.workunit.client.0.smithi145.stdout: "removed_snaps_queue": [], 2020-03-11T22:47:19.711 INFO:tasks.workunit.client.0.smithi145.stdout: "new_removed_snaps": [], 2020-03-11T22:47:19.711 INFO:tasks.workunit.client.0.smithi145.stdout: "new_purged_snaps": [], 2020-03-11T22:47:19.711 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_node_flags": {}, 2020-03-11T22:47:19.711 INFO:tasks.workunit.client.0.smithi145.stdout: "device_class_flags": {} 2020-03-11T22:47:19.711 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.712 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_metadata": [ 2020-03-11T22:47:19.712 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.712 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2020-03-11T22:47:19.712 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2020-03-11T22:47:19.712 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6804/48751,v1:127.0.0.1:6805/48751]", 2020-03-11T22:47:19.712 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2020-03-11T22:47:19.712 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2020-03-11T22:47:19.713 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:47:19.713 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:47:19.713 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:47:19.713 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:47:19.713 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:47:19.713 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:47:19.713 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:47:19.714 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:47:19.714 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:47:19.714 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:47:19.714 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:47:19.714 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:47:19.714 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "octopus", 2020-03-11T22:47:19.714 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:47:19.715 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:47:19.715 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:47:19.715 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2020-03-11T22:47:19.715 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XLDP", 2020-03-11T22:47:19.715 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:47:19.715 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2020-03-11T22:47:19.715 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2020-03-11T22:47:19.716 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:47:19.716 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2020-03-11T22:47:19.716 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6802/48751,v1:127.0.0.1:6803/48751]", 2020-03-11T22:47:19.716 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2020-03-11T22:47:19.716 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/48751,v1:127.0.0.1:6809/48751]", 2020-03-11T22:47:19.716 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/48751,v1:127.0.0.1:6807/48751]", 2020-03-11T22:47:19.716 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2020-03-11T22:47:19.717 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2020-03-11T22:47:19.717 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:47:19.717 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:47:19.717 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2020-03-11T22:47:19.717 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32655820", 2020-03-11T22:47:19.717 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:47:19.717 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:47:19.718 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2020-03-11T22:47:19.718 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:47:19.718 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:47:19.718 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2020-03-11T22:47:19.718 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.718 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.718 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2020-03-11T22:47:19.719 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2020-03-11T22:47:19.719 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6812/43891,v1:127.0.0.1:6813/43891]", 2020-03-11T22:47:19.719 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2020-03-11T22:47:19.719 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2020-03-11T22:47:19.719 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:47:19.719 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:47:19.719 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:47:19.719 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:47:19.720 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:47:19.720 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:47:19.720 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:47:19.720 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:47:19.720 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:47:19.720 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:47:19.720 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:47:19.721 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:47:19.721 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "octopus", 2020-03-11T22:47:19.721 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:47:19.721 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:47:19.721 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:47:19.721 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2020-03-11T22:47:19.721 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XLDP", 2020-03-11T22:47:19.722 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:47:19.722 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2020-03-11T22:47:19.722 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2020-03-11T22:47:19.722 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:47:19.722 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2020-03-11T22:47:19.722 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6810/43891,v1:127.0.0.1:6811/43891]", 2020-03-11T22:47:19.722 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2020-03-11T22:47:19.723 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/43891,v1:127.0.0.1:6817/43891]", 2020-03-11T22:47:19.723 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/43891,v1:127.0.0.1:6815/43891]", 2020-03-11T22:47:19.723 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2020-03-11T22:47:19.723 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2020-03-11T22:47:19.723 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:47:19.723 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:47:19.723 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2020-03-11T22:47:19.723 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32655820", 2020-03-11T22:47:19.724 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:47:19.724 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:47:19.724 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2020-03-11T22:47:19.724 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:47:19.724 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:47:19.724 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2020-03-11T22:47:19.724 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.725 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.725 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2020-03-11T22:47:19.725 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2020-03-11T22:47:19.725 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6812/48945,v1:127.0.0.1:6813/48945]", 2020-03-11T22:47:19.725 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2020-03-11T22:47:19.725 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2020-03-11T22:47:19.726 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:47:19.726 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:47:19.726 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:47:19.726 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:47:19.726 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:47:19.726 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:47:19.726 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:47:19.726 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:47:19.727 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:47:19.727 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:47:19.727 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:47:19.727 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:47:19.727 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "octopus", 2020-03-11T22:47:19.727 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:47:19.727 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:47:19.728 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:47:19.728 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2020-03-11T22:47:19.728 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XLDP", 2020-03-11T22:47:19.728 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:47:19.728 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2020-03-11T22:47:19.728 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2020-03-11T22:47:19.728 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:47:19.729 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2020-03-11T22:47:19.729 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6810/48945,v1:127.0.0.1:6811/48945]", 2020-03-11T22:47:19.729 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2020-03-11T22:47:19.729 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/48945,v1:127.0.0.1:6817/48945]", 2020-03-11T22:47:19.729 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/48945,v1:127.0.0.1:6815/48945]", 2020-03-11T22:47:19.729 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2020-03-11T22:47:19.729 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2020-03-11T22:47:19.730 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:47:19.730 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:47:19.730 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2020-03-11T22:47:19.730 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32655820", 2020-03-11T22:47:19.730 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:47:19.730 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:47:19.730 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2020-03-11T22:47:19.730 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:47:19.731 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:47:19.731 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2020-03-11T22:47:19.731 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:19.731 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:47:19.731 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap_first_committed": 1, 2020-03-11T22:47:19.731 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap_last_committed": 41, 2020-03-11T22:47:19.731 INFO:tasks.workunit.client.0.smithi145.stdout: "crushmap": { 2020-03-11T22:47:19.731 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": [ 2020-03-11T22:47:19.732 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.732 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2020-03-11T22:47:19.732 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.0", 2020-03-11T22:47:19.732 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2020-03-11T22:47:19.732 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.732 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.733 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2020-03-11T22:47:19.733 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.1", 2020-03-11T22:47:19.733 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2020-03-11T22:47:19.733 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.733 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.733 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2020-03-11T22:47:19.734 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.2", 2020-03-11T22:47:19.734 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2020-03-11T22:47:19.734 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:19.734 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:47:19.734 INFO:tasks.workunit.client.0.smithi145.stdout: "types": [ 2020-03-11T22:47:19.734 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.734 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 0, 2020-03-11T22:47:19.735 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd" 2020-03-11T22:47:19.735 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.735 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.735 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2020-03-11T22:47:19.735 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "host" 2020-03-11T22:47:19.735 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.735 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.736 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 2, 2020-03-11T22:47:19.736 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "chassis" 2020-03-11T22:47:19.736 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.736 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.736 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 3, 2020-03-11T22:47:19.736 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "rack" 2020-03-11T22:47:19.736 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.736 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.736 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 4, 2020-03-11T22:47:19.737 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "row" 2020-03-11T22:47:19.737 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.737 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.737 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 5, 2020-03-11T22:47:19.737 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "pdu" 2020-03-11T22:47:19.737 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.737 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.737 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 6, 2020-03-11T22:47:19.738 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "pod" 2020-03-11T22:47:19.738 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.738 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.738 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 7, 2020-03-11T22:47:19.738 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "room" 2020-03-11T22:47:19.738 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.738 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.739 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 8, 2020-03-11T22:47:19.739 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "datacenter" 2020-03-11T22:47:19.739 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.739 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.739 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 9, 2020-03-11T22:47:19.739 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "zone" 2020-03-11T22:47:19.740 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.740 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.740 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 10, 2020-03-11T22:47:19.740 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "region" 2020-03-11T22:47:19.740 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.740 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.740 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2020-03-11T22:47:19.741 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "root" 2020-03-11T22:47:19.741 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:19.741 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:47:19.741 INFO:tasks.workunit.client.0.smithi145.stdout: "buckets": [ 2020-03-11T22:47:19.741 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.741 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -1, 2020-03-11T22:47:19.741 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "default", 2020-03-11T22:47:19.742 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2020-03-11T22:47:19.742 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "root", 2020-03-11T22:47:19.742 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2020-03-11T22:47:19.742 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2020-03-11T22:47:19.742 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2020-03-11T22:47:19.743 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:47:19.743 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.743 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -3, 2020-03-11T22:47:19.743 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2020-03-11T22:47:19.743 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2020-03-11T22:47:19.743 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:19.743 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:19.744 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.744 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.744 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -2, 2020-03-11T22:47:19.744 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "default~hdd", 2020-03-11T22:47:19.744 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2020-03-11T22:47:19.744 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "root", 2020-03-11T22:47:19.744 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2020-03-11T22:47:19.745 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2020-03-11T22:47:19.745 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2020-03-11T22:47:19.745 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:47:19.745 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.745 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -4, 2020-03-11T22:47:19.745 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2020-03-11T22:47:19.745 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2020-03-11T22:47:19.745 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:19.746 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:19.746 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.746 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.746 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -3, 2020-03-11T22:47:19.746 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "smithi145", 2020-03-11T22:47:19.746 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2020-03-11T22:47:19.746 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "host", 2020-03-11T22:47:19.747 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2020-03-11T22:47:19.747 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2020-03-11T22:47:19.747 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2020-03-11T22:47:19.747 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:47:19.747 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.747 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2020-03-11T22:47:19.747 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2020-03-11T22:47:19.748 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2020-03-11T22:47:19.748 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.748 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.748 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2020-03-11T22:47:19.748 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2020-03-11T22:47:19.748 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 1 2020-03-11T22:47:19.748 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.748 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.749 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2020-03-11T22:47:19.749 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2020-03-11T22:47:19.749 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 2 2020-03-11T22:47:19.749 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:19.749 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:19.749 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.749 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.750 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -4, 2020-03-11T22:47:19.750 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "smithi145~hdd", 2020-03-11T22:47:19.750 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2020-03-11T22:47:19.750 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "host", 2020-03-11T22:47:19.750 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2020-03-11T22:47:19.750 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2020-03-11T22:47:19.750 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2020-03-11T22:47:19.751 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:47:19.751 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.751 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2020-03-11T22:47:19.751 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2020-03-11T22:47:19.751 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2020-03-11T22:47:19.751 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.752 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.752 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2020-03-11T22:47:19.752 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2020-03-11T22:47:19.752 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 1 2020-03-11T22:47:19.752 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.752 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.753 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2020-03-11T22:47:19.753 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2020-03-11T22:47:19.753 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 2 2020-03-11T22:47:19.753 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:19.753 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:19.753 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:19.753 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:47:19.754 INFO:tasks.workunit.client.0.smithi145.stdout: "rules": [ 2020-03-11T22:47:19.754 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.754 INFO:tasks.workunit.client.0.smithi145.stdout: "rule_id": 0, 2020-03-11T22:47:19.754 INFO:tasks.workunit.client.0.smithi145.stdout: "rule_name": "replicated_rule", 2020-03-11T22:47:19.754 INFO:tasks.workunit.client.0.smithi145.stdout: "ruleset": 0, 2020-03-11T22:47:19.754 INFO:tasks.workunit.client.0.smithi145.stdout: "type": 1, 2020-03-11T22:47:19.754 INFO:tasks.workunit.client.0.smithi145.stdout: "min_size": 1, 2020-03-11T22:47:19.754 INFO:tasks.workunit.client.0.smithi145.stdout: "max_size": 10, 2020-03-11T22:47:19.755 INFO:tasks.workunit.client.0.smithi145.stdout: "steps": [ 2020-03-11T22:47:19.755 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.755 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "take", 2020-03-11T22:47:19.755 INFO:tasks.workunit.client.0.smithi145.stdout: "item": -1, 2020-03-11T22:47:19.755 INFO:tasks.workunit.client.0.smithi145.stdout: "item_name": "default" 2020-03-11T22:47:19.755 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.755 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.756 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "choose_firstn", 2020-03-11T22:47:19.756 INFO:tasks.workunit.client.0.smithi145.stdout: "num": 0, 2020-03-11T22:47:19.756 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "osd" 2020-03-11T22:47:19.756 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.756 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.756 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "emit" 2020-03-11T22:47:19.757 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:19.757 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:19.757 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.757 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.757 INFO:tasks.workunit.client.0.smithi145.stdout: "rule_id": 1, 2020-03-11T22:47:19.757 INFO:tasks.workunit.client.0.smithi145.stdout: "rule_name": "test", 2020-03-11T22:47:19.758 INFO:tasks.workunit.client.0.smithi145.stdout: "ruleset": 1, 2020-03-11T22:47:19.758 INFO:tasks.workunit.client.0.smithi145.stdout: "type": 3, 2020-03-11T22:47:19.758 INFO:tasks.workunit.client.0.smithi145.stdout: "min_size": 3, 2020-03-11T22:47:19.758 INFO:tasks.workunit.client.0.smithi145.stdout: "max_size": 3, 2020-03-11T22:47:19.758 INFO:tasks.workunit.client.0.smithi145.stdout: "steps": [ 2020-03-11T22:47:19.758 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.758 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "set_chooseleaf_tries", 2020-03-11T22:47:19.759 INFO:tasks.workunit.client.0.smithi145.stdout: "num": 5 2020-03-11T22:47:19.759 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.759 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.759 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "set_choose_tries", 2020-03-11T22:47:19.759 INFO:tasks.workunit.client.0.smithi145.stdout: "num": 100 2020-03-11T22:47:19.759 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.759 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.759 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "take", 2020-03-11T22:47:19.760 INFO:tasks.workunit.client.0.smithi145.stdout: "item": -1, 2020-03-11T22:47:19.760 INFO:tasks.workunit.client.0.smithi145.stdout: "item_name": "default" 2020-03-11T22:47:19.760 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.760 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.760 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "choose_indep", 2020-03-11T22:47:19.760 INFO:tasks.workunit.client.0.smithi145.stdout: "num": 0, 2020-03-11T22:47:19.760 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "osd" 2020-03-11T22:47:19.761 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.761 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.761 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "emit" 2020-03-11T22:47:19.761 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:19.761 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:19.761 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:19.761 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:47:19.761 INFO:tasks.workunit.client.0.smithi145.stdout: "tunables": { 2020-03-11T22:47:19.762 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_local_tries": 0, 2020-03-11T22:47:19.762 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:47:19.762 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_total_tries": 50, 2020-03-11T22:47:19.762 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:47:19.762 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:47:19.762 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_stable": 1, 2020-03-11T22:47:19.762 INFO:tasks.workunit.client.0.smithi145.stdout: "straw_calc_version": 1, 2020-03-11T22:47:19.763 INFO:tasks.workunit.client.0.smithi145.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:47:19.763 INFO:tasks.workunit.client.0.smithi145.stdout: "profile": "jewel", 2020-03-11T22:47:19.763 INFO:tasks.workunit.client.0.smithi145.stdout: "optimal_tunables": 1, 2020-03-11T22:47:19.763 INFO:tasks.workunit.client.0.smithi145.stdout: "legacy_tunables": 0, 2020-03-11T22:47:19.763 INFO:tasks.workunit.client.0.smithi145.stdout: "minimum_required_version": "jewel", 2020-03-11T22:47:19.763 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables": 1, 2020-03-11T22:47:19.763 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables2": 1, 2020-03-11T22:47:19.764 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v2_rules": 1, 2020-03-11T22:47:19.764 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables3": 1, 2020-03-11T22:47:19.764 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v3_rules": 0, 2020-03-11T22:47:19.764 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v4_buckets": 1, 2020-03-11T22:47:19.764 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables5": 1, 2020-03-11T22:47:19.764 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v5_rules": 0 2020-03-11T22:47:19.764 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.764 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_args": {} 2020-03-11T22:47:19.765 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.765 INFO:tasks.workunit.client.0.smithi145.stdout: "fsmap": { 2020-03-11T22:47:19.765 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 1, 2020-03-11T22:47:19.765 INFO:tasks.workunit.client.0.smithi145.stdout: "default_fscid": -1, 2020-03-11T22:47:19.765 INFO:tasks.workunit.client.0.smithi145.stdout: "compat": { 2020-03-11T22:47:19.765 INFO:tasks.workunit.client.0.smithi145.stdout: "compat": {}, 2020-03-11T22:47:19.765 INFO:tasks.workunit.client.0.smithi145.stdout: "ro_compat": {}, 2020-03-11T22:47:19.766 INFO:tasks.workunit.client.0.smithi145.stdout: "incompat": { 2020-03-11T22:47:19.766 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_1": "base v0.20", 2020-03-11T22:47:19.766 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:47:19.766 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:47:19.766 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:47:19.766 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:47:19.766 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:47:19.766 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_8": "no anchor table", 2020-03-11T22:47:19.766 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_9": "file layout v2", 2020-03-11T22:47:19.767 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:47:19.767 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:19.767 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.767 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_flags": { 2020-03-11T22:47:19.767 INFO:tasks.workunit.client.0.smithi145.stdout: "enable_multiple": false, 2020-03-11T22:47:19.767 INFO:tasks.workunit.client.0.smithi145.stdout: "ever_enabled_multiple": false 2020-03-11T22:47:19.767 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.767 INFO:tasks.workunit.client.0.smithi145.stdout: "standbys": [], 2020-03-11T22:47:19.768 INFO:tasks.workunit.client.0.smithi145.stdout: "filesystems": [] 2020-03-11T22:47:19.768 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.768 INFO:tasks.workunit.client.0.smithi145.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:47:19.768 INFO:tasks.workunit.client.0.smithi145.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:47:19.768 INFO:tasks.workunit.client.0.smithi145.stdout: "auth": { 2020-03-11T22:47:19.768 INFO:tasks.workunit.client.0.smithi145.stdout: "first_committed": 1, 2020-03-11T22:47:19.768 INFO:tasks.workunit.client.0.smithi145.stdout: "last_committed": 5, 2020-03-11T22:47:19.769 INFO:tasks.workunit.client.0.smithi145.stdout: "num_secrets": 3 2020-03-11T22:47:19.769 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.769 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg": 1, 2020-03-11T22:47:19.769 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_active": 1, 2020-03-11T22:47:19.769 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_unknown": 0, 2020-03-11T22:47:19.769 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osd": 3, 2020-03-11T22:47:19.769 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_sum": { 2020-03-11T22:47:19.769 INFO:tasks.workunit.client.0.smithi145.stdout: "stat_sum": { 2020-03-11T22:47:19.770 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes": 48430, 2020-03-11T22:47:19.770 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects": 101, 2020-03-11T22:47:19.770 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_clones": 1, 2020-03-11T22:47:19.770 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_copies": 303, 2020-03-11T22:47:19.770 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:47:19.770 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing": 0, 2020-03-11T22:47:19.770 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_degraded": 101, 2020-03-11T22:47:19.770 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_misplaced": 0, 2020-03-11T22:47:19.771 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_unfound": 0, 2020-03-11T22:47:19.771 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_dirty": 101, 2020-03-11T22:47:19.771 INFO:tasks.workunit.client.0.smithi145.stdout: "num_whiteouts": 0, 2020-03-11T22:47:19.771 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read": 0, 2020-03-11T22:47:19.771 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read_kb": 0, 2020-03-11T22:47:19.771 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write": 101, 2020-03-11T22:47:19.771 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write_kb": 101, 2020-03-11T22:47:19.771 INFO:tasks.workunit.client.0.smithi145.stdout: "num_scrub_errors": 0, 2020-03-11T22:47:19.772 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:47:19.772 INFO:tasks.workunit.client.0.smithi145.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:47:19.772 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_recovered": 0, 2020-03-11T22:47:19.772 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_recovered": 0, 2020-03-11T22:47:19.772 INFO:tasks.workunit.client.0.smithi145.stdout: "num_keys_recovered": 0, 2020-03-11T22:47:19.772 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_omap": 0, 2020-03-11T22:47:19.772 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:47:19.773 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:47:19.773 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush": 0, 2020-03-11T22:47:19.773 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_kb": 0, 2020-03-11T22:47:19.773 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict": 0, 2020-03-11T22:47:19.773 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_kb": 0, 2020-03-11T22:47:19.773 INFO:tasks.workunit.client.0.smithi145.stdout: "num_promote": 0, 2020-03-11T22:47:19.773 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_high": 0, 2020-03-11T22:47:19.774 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_low": 0, 2020-03-11T22:47:19.774 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_some": 0, 2020-03-11T22:47:19.774 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_full": 0, 2020-03-11T22:47:19.774 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_pinned": 0, 2020-03-11T22:47:19.774 INFO:tasks.workunit.client.0.smithi145.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:47:19.774 INFO:tasks.workunit.client.0.smithi145.stdout: "num_large_omap_objects": 0, 2020-03-11T22:47:19.774 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_manifest": 0, 2020-03-11T22:47:19.774 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_bytes": 0, 2020-03-11T22:47:19.775 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_keys": 0, 2020-03-11T22:47:19.775 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_repaired": 0 2020-03-11T22:47:19.775 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.775 INFO:tasks.workunit.client.0.smithi145.stdout: "store_stats": { 2020-03-11T22:47:19.775 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 0, 2020-03-11T22:47:19.775 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 0, 2020-03-11T22:47:19.775 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2020-03-11T22:47:19.775 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 0, 2020-03-11T22:47:19.776 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 0, 2020-03-11T22:47:19.776 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2020-03-11T22:47:19.776 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2020-03-11T22:47:19.776 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2020-03-11T22:47:19.776 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 0, 2020-03-11T22:47:19.776 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 0 2020-03-11T22:47:19.776 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.776 INFO:tasks.workunit.client.0.smithi145.stdout: "log_size": 18, 2020-03-11T22:47:19.777 INFO:tasks.workunit.client.0.smithi145.stdout: "ondisk_log_size": 18, 2020-03-11T22:47:19.777 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 3, 2020-03-11T22:47:19.777 INFO:tasks.workunit.client.0.smithi145.stdout: "acting": 3, 2020-03-11T22:47:19.777 INFO:tasks.workunit.client.0.smithi145.stdout: "num_store_stats": 0 2020-03-11T22:47:19.777 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.777 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_sum": { 2020-03-11T22:47:19.777 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 0, 2020-03-11T22:47:19.777 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 0, 2020-03-11T22:47:19.777 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pgs": 3, 2020-03-11T22:47:19.778 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osds": 3, 2020-03-11T22:47:19.778 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_osds": 3, 2020-03-11T22:47:19.778 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:47:19.778 INFO:tasks.workunit.client.0.smithi145.stdout: "kb": 314572800, 2020-03-11T22:47:19.778 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used": 3180032, 2020-03-11T22:47:19.778 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_data": 33984, 2020-03-11T22:47:19.778 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_omap": 11, 2020-03-11T22:47:19.779 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_meta": 3145716, 2020-03-11T22:47:19.779 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_avail": 311392768, 2020-03-11T22:47:19.779 INFO:tasks.workunit.client.0.smithi145.stdout: "statfs": { 2020-03-11T22:47:19.779 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 322122547200, 2020-03-11T22:47:19.779 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 318866194432, 2020-03-11T22:47:19.779 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 3221225472, 2020-03-11T22:47:19.779 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 34799616, 2020-03-11T22:47:19.780 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 1529153, 2020-03-11T22:47:19.780 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2020-03-11T22:47:19.780 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2020-03-11T22:47:19.780 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2020-03-11T22:47:19.780 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 11530, 2020-03-11T22:47:19.780 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 3221213942 2020-03-11T22:47:19.780 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.781 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_peers": [], 2020-03-11T22:47:19.781 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:47:19.781 INFO:tasks.workunit.client.0.smithi145.stdout: "num_snap_trimming": 0, 2020-03-11T22:47:19.781 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shards_repaired": 0, 2020-03-11T22:47:19.781 INFO:tasks.workunit.client.0.smithi145.stdout: "op_queue_age_hist": { 2020-03-11T22:47:19.781 INFO:tasks.workunit.client.0.smithi145.stdout: "histogram": [], 2020-03-11T22:47:19.781 INFO:tasks.workunit.client.0.smithi145.stdout: "upper_bound": 1 2020-03-11T22:47:19.781 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.782 INFO:tasks.workunit.client.0.smithi145.stdout: "perf_stat": { 2020-03-11T22:47:19.782 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ms": 0, 2020-03-11T22:47:19.782 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ms": 0, 2020-03-11T22:47:19.782 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ns": 0, 2020-03-11T22:47:19.782 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ns": 0 2020-03-11T22:47:19.782 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.782 INFO:tasks.workunit.client.0.smithi145.stdout: "alerts": [], 2020-03-11T22:47:19.783 INFO:tasks.workunit.client.0.smithi145.stdout: "network_ping_times": [] 2020-03-11T22:47:19.783 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.783 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_sum_by_class": { 2020-03-11T22:47:19.783 INFO:tasks.workunit.client.0.smithi145.stdout: "hdd": { 2020-03-11T22:47:19.783 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 0, 2020-03-11T22:47:19.783 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 0, 2020-03-11T22:47:19.783 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pgs": 3, 2020-03-11T22:47:19.784 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osds": 3, 2020-03-11T22:47:19.784 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_osds": 3, 2020-03-11T22:47:19.784 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:47:19.784 INFO:tasks.workunit.client.0.smithi145.stdout: "kb": 314572800, 2020-03-11T22:47:19.785 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used": 3180032, 2020-03-11T22:47:19.785 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_data": 33984, 2020-03-11T22:47:19.785 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_omap": 11, 2020-03-11T22:47:19.785 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_meta": 3145716, 2020-03-11T22:47:19.785 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_avail": 311392768, 2020-03-11T22:47:19.785 INFO:tasks.workunit.client.0.smithi145.stdout: "statfs": { 2020-03-11T22:47:19.785 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 322122547200, 2020-03-11T22:47:19.785 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 318866194432, 2020-03-11T22:47:19.786 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 3221225472, 2020-03-11T22:47:19.786 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 34799616, 2020-03-11T22:47:19.786 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 1529153, 2020-03-11T22:47:19.786 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2020-03-11T22:47:19.786 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2020-03-11T22:47:19.786 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2020-03-11T22:47:19.786 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 11530, 2020-03-11T22:47:19.787 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 3221213942 2020-03-11T22:47:19.787 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.787 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_peers": [], 2020-03-11T22:47:19.787 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:47:19.787 INFO:tasks.workunit.client.0.smithi145.stdout: "num_snap_trimming": 0, 2020-03-11T22:47:19.787 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shards_repaired": 0, 2020-03-11T22:47:19.787 INFO:tasks.workunit.client.0.smithi145.stdout: "op_queue_age_hist": { 2020-03-11T22:47:19.788 INFO:tasks.workunit.client.0.smithi145.stdout: "histogram": [], 2020-03-11T22:47:19.788 INFO:tasks.workunit.client.0.smithi145.stdout: "upper_bound": 1 2020-03-11T22:47:19.788 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.788 INFO:tasks.workunit.client.0.smithi145.stdout: "perf_stat": { 2020-03-11T22:47:19.788 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ms": 0, 2020-03-11T22:47:19.788 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ms": 0, 2020-03-11T22:47:19.788 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ns": 0, 2020-03-11T22:47:19.789 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ns": 0 2020-03-11T22:47:19.789 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.789 INFO:tasks.workunit.client.0.smithi145.stdout: "alerts": [], 2020-03-11T22:47:19.789 INFO:tasks.workunit.client.0.smithi145.stdout: "network_ping_times": [] 2020-03-11T22:47:19.789 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:19.790 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.790 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_stats": [ 2020-03-11T22:47:19.790 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.790 INFO:tasks.workunit.client.0.smithi145.stdout: "poolid": 1, 2020-03-11T22:47:19.790 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg": 1, 2020-03-11T22:47:19.790 INFO:tasks.workunit.client.0.smithi145.stdout: "stat_sum": { 2020-03-11T22:47:19.790 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes": 48430, 2020-03-11T22:47:19.791 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects": 101, 2020-03-11T22:47:19.791 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_clones": 1, 2020-03-11T22:47:19.791 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_copies": 303, 2020-03-11T22:47:19.791 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:47:19.791 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing": 0, 2020-03-11T22:47:19.791 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_degraded": 101, 2020-03-11T22:47:19.791 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_misplaced": 0, 2020-03-11T22:47:19.792 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_unfound": 0, 2020-03-11T22:47:19.792 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_dirty": 101, 2020-03-11T22:47:19.792 INFO:tasks.workunit.client.0.smithi145.stdout: "num_whiteouts": 0, 2020-03-11T22:47:19.792 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read": 0, 2020-03-11T22:47:19.792 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read_kb": 0, 2020-03-11T22:47:19.792 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write": 101, 2020-03-11T22:47:19.792 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write_kb": 101, 2020-03-11T22:47:19.793 INFO:tasks.workunit.client.0.smithi145.stdout: "num_scrub_errors": 0, 2020-03-11T22:47:19.793 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:47:19.793 INFO:tasks.workunit.client.0.smithi145.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:47:19.793 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_recovered": 0, 2020-03-11T22:47:19.793 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_recovered": 0, 2020-03-11T22:47:19.793 INFO:tasks.workunit.client.0.smithi145.stdout: "num_keys_recovered": 0, 2020-03-11T22:47:19.793 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_omap": 0, 2020-03-11T22:47:19.794 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:47:19.794 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:47:19.794 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush": 0, 2020-03-11T22:47:19.794 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_kb": 0, 2020-03-11T22:47:19.794 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict": 0, 2020-03-11T22:47:19.794 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_kb": 0, 2020-03-11T22:47:19.794 INFO:tasks.workunit.client.0.smithi145.stdout: "num_promote": 0, 2020-03-11T22:47:19.795 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_high": 0, 2020-03-11T22:47:19.795 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_low": 0, 2020-03-11T22:47:19.795 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_some": 0, 2020-03-11T22:47:19.795 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_full": 0, 2020-03-11T22:47:19.795 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_pinned": 0, 2020-03-11T22:47:19.795 INFO:tasks.workunit.client.0.smithi145.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:47:19.795 INFO:tasks.workunit.client.0.smithi145.stdout: "num_large_omap_objects": 0, 2020-03-11T22:47:19.796 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_manifest": 0, 2020-03-11T22:47:19.796 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_bytes": 0, 2020-03-11T22:47:19.796 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_keys": 0, 2020-03-11T22:47:19.796 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_repaired": 0 2020-03-11T22:47:19.796 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.796 INFO:tasks.workunit.client.0.smithi145.stdout: "store_stats": { 2020-03-11T22:47:19.796 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 0, 2020-03-11T22:47:19.796 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 0, 2020-03-11T22:47:19.797 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2020-03-11T22:47:19.797 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 19857408, 2020-03-11T22:47:19.797 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 1241088, 2020-03-11T22:47:19.797 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2020-03-11T22:47:19.797 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2020-03-11T22:47:19.797 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2020-03-11T22:47:19.797 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 0, 2020-03-11T22:47:19.798 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 0 2020-03-11T22:47:19.798 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.798 INFO:tasks.workunit.client.0.smithi145.stdout: "log_size": 18, 2020-03-11T22:47:19.798 INFO:tasks.workunit.client.0.smithi145.stdout: "ondisk_log_size": 18, 2020-03-11T22:47:19.798 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 3, 2020-03-11T22:47:19.798 INFO:tasks.workunit.client.0.smithi145.stdout: "acting": 3, 2020-03-11T22:47:19.799 INFO:tasks.workunit.client.0.smithi145.stdout: "num_store_stats": 3 2020-03-11T22:47:19.799 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:19.799 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:47:19.799 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_stats": [ 2020-03-11T22:47:19.799 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.799 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2020-03-11T22:47:19.800 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 150323855373 2020-03-11T22:47:19.800 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.800 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.800 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2020-03-11T22:47:19.800 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 42949672980 2020-03-11T22:47:19.800 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.801 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.801 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2020-03-11T22:47:19.801 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 154618822669 2020-03-11T22:47:19.801 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:19.801 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:47:19.801 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_by_state": [ 2020-03-11T22:47:19.802 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.802 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "active+undersized+degraded", 2020-03-11T22:47:19.802 INFO:tasks.workunit.client.0.smithi145.stdout: "num": 1 2020-03-11T22:47:19.802 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:19.802 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:47:19.802 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_by_osd": [ 2020-03-11T22:47:19.803 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.803 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2020-03-11T22:47:19.803 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 0, 2020-03-11T22:47:19.803 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 1, 2020-03-11T22:47:19.803 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:47:19.803 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.803 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.804 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2147483647, 2020-03-11T22:47:19.804 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 0, 2020-03-11T22:47:19.804 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 1, 2020-03-11T22:47:19.804 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:47:19.804 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.804 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.805 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2020-03-11T22:47:19.805 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 0, 2020-03-11T22:47:19.805 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 0, 2020-03-11T22:47:19.805 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:47:19.805 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.806 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.806 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2020-03-11T22:47:19.807 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 1, 2020-03-11T22:47:19.807 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 1, 2020-03-11T22:47:19.807 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:47:19.807 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:19.807 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:47:19.807 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snaps": [ 2020-03-11T22:47:19.807 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:19.808 INFO:tasks.workunit.client.0.smithi145.stdout: "pool": 1, 2020-03-11T22:47:19.808 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snaps": {} 2020-03-11T22:47:19.808 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:19.808 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:47:19.808 INFO:tasks.workunit.client.0.smithi145.stdout: "servicemap": { 2020-03-11T22:47:19.808 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 1, 2020-03-11T22:47:19.808 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2020-03-11T22:45:16.868571+0000", 2020-03-11T22:47:19.809 INFO:tasks.workunit.client.0.smithi145.stdout: "services": {} 2020-03-11T22:47:19.809 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:19.809 INFO:tasks.workunit.client.0.smithi145.stdout: "paxos": { 2020-03-11T22:47:19.809 INFO:tasks.workunit.client.0.smithi145.stdout: "first_committed": 1, 2020-03-11T22:47:19.809 INFO:tasks.workunit.client.0.smithi145.stdout: "last_committed": 353, 2020-03-11T22:47:19.809 INFO:tasks.workunit.client.0.smithi145.stdout: "last_pn": 0, 2020-03-11T22:47:19.810 INFO:tasks.workunit.client.0.smithi145.stdout: "accepted_pn": 0 2020-03-11T22:47:19.810 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:19.810 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:47:19.812 INFO:tasks.workunit.client.0.smithi145.stderr:report 814152439 2020-03-11T22:47:19.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:47:19.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:47:19.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.0 2020-03-11T22:47:19.812 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:47:19.812 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:47:19.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:47:19.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:47:19.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:47:19.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:47:19.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 0 --op log --pgid 1.0 2020-03-11T22:47:19.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:47:19.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:47:19.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=0 2020-03-11T22:47:19.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:47:19.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/0 2020-03-11T22:47:19.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/0 --op log --pgid 1.0 2020-03-11T22:47:20.310 INFO:tasks.workunit.client.0.smithi145.stderr:PG '1.0' not found 2020-03-11T22:47:20.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: return 1 2020-03-11T22:47:20.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 0 2020-03-11T22:47:20.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:47:20.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:47:20.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:47:20.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:47:20.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:47:20.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:47:20.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:47:20.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:47:20.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:47:20.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:47:20.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:47:20.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:47:20.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:47:20.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:47:20.832 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:47:20.832 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:47:20.832 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:47:20.832 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:47:20.832 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:20.832 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:47:20.833 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:47:20.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:47:20.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:47:20.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:47:20.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:47:20.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:47:20.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:47:20.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:20.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:47:20.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:47:20.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:47:20.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:47:20.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:20.835 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-03-11T22:47:20.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:47:20.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:47:20.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:47:20.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:47:20.836 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2020-03-11T22:47:20.860 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:47:20.856+0000 7f89718e1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:20.862 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:47:20.858+0000 7f89718e1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:20.864 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:47:20.861+0000 7f89718e1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:21.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:47:21.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:47:21.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:47:21.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:47:21.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:47:21.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:47:21.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:47:21.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:47:21.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:47:21.198 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:47:21.382 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:47:21.378+0000 7f89718e1ec0 -1 Falling back to public interface 2020-03-11T22:47:21.543 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 35 up_thru 40 down_at 30 last_clean_interval [5,28) [v2:127.0.0.1:6802/48751,v1:127.0.0.1:6803/48751] [v2:127.0.0.1:6804/48751,v1:127.0.0.1:6805/48751] exists,up 04216e1d-6638-4e33-8cad-1ea8a9c30138 2020-03-11T22:47:21.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:47:21.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:47:21.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:47:21.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:47:21.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.2 2020-03-11T22:47:21.544 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:47:21.544 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:47:21.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:47:21.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:47:21.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:47:21.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:47:21.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 2 --op log --pgid 1.0 2020-03-11T22:47:21.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:47:21.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:47:21.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=2 2020-03-11T22:47:21.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:47:21.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/2 2020-03-11T22:47:21.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/2 --op log --pgid 1.0 2020-03-11T22:47:22.222 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:47:22.219+0000 7f89718e1ec0 -1 osd.0 41 log_to_monitors {default=true} 2020-03-11T22:47:22.417 INFO:tasks.workunit.client.0.smithi145.stderr:PG '1.0' not found 2020-03-11T22:47:22.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: return 1 2020-03-11T22:47:22.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 2 2020-03-11T22:47:22.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:47:22.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:47:22.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:47:22.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:47:22.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:47:22.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:47:22.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:47:22.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:47:22.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:47:22.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:47:22.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:47:22.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:47:22.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:47:22.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:47:22.941 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:47:22.941 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:47:22.941 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:47:22.941 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:47:22.942 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:22.942 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:47:22.942 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:47:22.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:47:22.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:47:22.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:47:22.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:47:22.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:47:22.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:47:22.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:22.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:47:22.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:47:22.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:47:22.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:47:22.944 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2020-03-11T22:47:22.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:22.945 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-03-11T22:47:22.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:47:22.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:47:22.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:47:22.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:47:22.968 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:47:22.965+0000 7f3d9b515ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:22.970 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:47:22.967+0000 7f3d9b515ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:22.971 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:47:22.969+0000 7f3d9b515ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:23.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:47:23.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:47:23.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:47:23.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:47:23.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:47:23.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:47:23.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:47:23.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:47:23.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:47:23.327 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:47:23.488 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:47:23.486+0000 7f3d9b515ec0 -1 Falling back to public interface 2020-03-11T22:47:23.686 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 36 up_thru 0 down_at 32 last_clean_interval [15,28) [v2:127.0.0.1:6810/48945,v1:127.0.0.1:6811/48945] [v2:127.0.0.1:6812/48945,v1:127.0.0.1:6813/48945] exists,up 1df352e0-156e-4ee7-a96d-63bceae4b98b 2020-03-11T22:47:23.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:47:23.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:47:23.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:47:23.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:336: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 1 --op log --pgid 1.0 2020-03-11T22:47:23.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:47:23.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:47:23.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-03-11T22:47:23.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:47:23.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-03-11T22:47:23.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op log --pgid 1.0 2020-03-11T22:47:24.353 INFO:tasks.workunit.client.0.smithi145.stderr:PG '1.0' not found 2020-03-11T22:47:24.394 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:47:24.391+0000 7f3d9b515ec0 -1 osd.2 41 log_to_monitors {default=true} 2020-03-11T22:47:24.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: return 1 2020-03-11T22:47:24.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:47:24.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: wait_for_clean 2020-03-11T22:47:24.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:47:24.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:47:24.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:47:24.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:47:24.874 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:47:24.875 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:47:24.875 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:47:24.875 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:47:24.876 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:47:24.876 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:47:24.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:47:24.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:47:24.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:47:24.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:47:24.941 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:47:25.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:47:25.291 INFO:tasks.workunit.client.0.smithi145.stderr:1 2020-03-11T22:47:25.291 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2020-03-11T22:47:25.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:47:25.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:47:25.291 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:47:25.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=184683593731 2020-03-11T22:47:25.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-184683593731' 2020-03-11T22:47:25.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:47:25.423 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:47:25.512 INFO:tasks.workunit.client.0.smithi145.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:47:25.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:47:25.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-184683593731 1-' 2020-03-11T22:47:25.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:47:25.521 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:47:25.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=197568495619 2020-03-11T22:47:25.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-184683593731 1- 2-197568495619' 2020-03-11T22:47:25.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:47:25.650 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-184683593731 2020-03-11T22:47:25.650 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:47:25.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:47:25.651 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-184683593731 2020-03-11T22:47:25.651 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:47:25.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=184683593731 2020-03-11T22:47:25.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 184683593731' 2020-03-11T22:47:25.652 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:47:25.653 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 184683593731 2020-03-11T22:47:25.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 1 -lt 184683593731 2020-03-11T22:47:25.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:47:26.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:47:26.994 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:47:27.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 184683593731 -lt 184683593731 2020-03-11T22:47:27.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:47:27.346 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:47:27.346 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:47:27.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:47:27.347 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:47:27.347 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:47:27.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:47:27.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:47:27.348 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:47:27.349 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 2020-03-11T22:47:27.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672980 -lt 2020-03-11T22:47:27.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672980: unary operator expected 2020-03-11T22:47:27.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:47:27.694 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-197568495619 2020-03-11T22:47:27.694 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:47:27.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:47:27.695 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-197568495619 2020-03-11T22:47:27.695 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:47:27.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=197568495619 2020-03-11T22:47:27.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 197568495619' 2020-03-11T22:47:27.696 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:47:27.697 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 197568495619 2020-03-11T22:47:28.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 197568495619 -lt 197568495619 2020-03-11T22:47:28.036 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:47:28.037 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:28.037 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:28.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:47:28.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:28.479 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:28.479 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:28.480 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:28.480 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:28.480 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:28.480 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:28.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:28.808 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:28.808 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:28.808 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:29.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:47:29.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:47:29.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:47:29.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:47:29.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:29.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:29.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:29.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:29.348 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:29.348 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:29.349 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:29.349 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:29.349 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:29.349 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:29.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:29.683 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:29.683 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:29.683 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:30.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:47:30.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:30.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:30.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:30.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:30.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:30.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:30.123 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:30.123 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:30.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:47:30.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:47:30.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:47:30.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:30.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:47:30.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:30.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:30.766 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:30.766 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:30.766 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:30.766 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:30.766 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:30.767 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:31.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:31.100 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:31.100 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:31.100 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:31.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:47:31.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:31.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:31.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:31.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:31.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:31.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:31.551 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:31.551 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:31.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:47:31.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:47:31.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:47:31.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:31.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:47:32.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:32.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:32.399 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:32.399 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:32.399 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:32.399 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:32.400 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:32.400 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:32.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:32.731 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:32.732 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:32.732 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:33.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:47:33.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:33.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:33.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:33.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:33.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:33.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:33.202 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:33.202 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:33.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:47:33.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:47:33.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:47:33.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:33.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:47:34.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:34.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:34.444 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:34.445 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:34.445 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:34.445 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:34.445 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:34.445 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:34.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:34.778 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:34.779 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:34.779 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:35.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:47:35.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:35.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:35.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:35.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:35.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:35.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:35.225 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:35.225 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:35.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:47:35.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:47:35.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:47:35.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:35.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:47:37.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:37.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:37.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:37.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:37.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:37.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:37.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:37.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:37.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:37.628 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:37.628 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:37.628 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:38.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:47:38.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:38.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:38.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:38.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:38.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:38.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:38.073 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:38.074 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:38.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:47:38.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:47:38.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:47:38.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:38.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:47:41.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:41.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:41.714 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:41.714 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:41.715 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:41.715 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:41.715 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:41.715 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:42.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:42.058 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:42.058 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:42.058 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:42.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:47:42.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:42.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:42.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:42.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:42.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:42.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:42.502 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:42.502 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:42.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:47:42.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:47:42.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:47:42.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:42.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:47:49.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:49.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:49.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:49.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:49.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:49.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:49.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:49.345 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:49.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:49.689 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:49.689 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:49.689 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:50.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:47:50.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:50.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:50.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:50.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:50.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:50.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:50.130 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:50.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:50.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:47:50.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:47:50.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:47:50.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:50.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:47:57.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:57.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:57.883 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:57.883 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:57.883 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:57.884 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:57.884 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:57.884 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:58.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:58.223 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:58.223 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:58.223 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:58.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:47:58.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:58.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:58.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:58.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:58.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:58.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:58.673 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:58.673 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:59.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:47:59.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:47:59.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:47:59.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:47:59.557 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:47:59.558 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_fingerprint": "08e61c9d-a9f7-449a-83c3-d393604848c8", 2020-03-11T22:47:59.558 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:47:59.558 INFO:tasks.workunit.client.0.smithi145.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:47:59.558 INFO:tasks.workunit.client.0.smithi145.stdout: "timestamp": "2020-03-11T22:47:59.553895+0000", 2020-03-11T22:47:59.558 INFO:tasks.workunit.client.0.smithi145.stdout: "tag": "", 2020-03-11T22:47:59.558 INFO:tasks.workunit.client.0.smithi145.stdout: "health": { 2020-03-11T22:47:59.559 INFO:tasks.workunit.client.0.smithi145.stdout: "status": "HEALTH_WARN", 2020-03-11T22:47:59.559 INFO:tasks.workunit.client.0.smithi145.stdout: "checks": { 2020-03-11T22:47:59.559 INFO:tasks.workunit.client.0.smithi145.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:47:59.559 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:47:59.559 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2020-03-11T22:47:59.559 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:47:59.559 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 17 2020-03-11T22:47:59.559 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.560 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [], 2020-03-11T22:47:59.560 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2020-03-11T22:47:59.560 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.560 INFO:tasks.workunit.client.0.smithi145.stdout: "OSD_DOWN": { 2020-03-11T22:47:59.560 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:47:59.560 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2020-03-11T22:47:59.560 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "1 osds down", 2020-03-11T22:47:59.561 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 1 2020-03-11T22:47:59.561 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.561 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2020-03-11T22:47:59.561 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.561 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "osd.1 (root=default,host=smithi145) is down" 2020-03-11T22:47:59.561 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:59.561 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:47:59.562 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2020-03-11T22:47:59.562 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.562 INFO:tasks.workunit.client.0.smithi145.stdout: "PG_DEGRADED": { 2020-03-11T22:47:59.562 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:47:59.562 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2020-03-11T22:47:59.562 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2020-03-11T22:47:59.562 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 1 2020-03-11T22:47:59.562 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.563 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2020-03-11T22:47:59.563 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.563 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [2147483647,0,2]" 2020-03-11T22:47:59.563 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:59.563 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:47:59.563 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2020-03-11T22:47:59.563 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.564 INFO:tasks.workunit.client.0.smithi145.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:47:59.564 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:47:59.564 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2020-03-11T22:47:59.564 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:47:59.564 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 1 2020-03-11T22:47:59.564 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.564 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2020-03-11T22:47:59.565 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.565 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:47:59.565 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.565 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.565 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:47:59.565 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:59.565 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:47:59.565 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2020-03-11T22:47:59.566 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:59.566 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.567 INFO:tasks.workunit.client.0.smithi145.stdout: "mutes": [] 2020-03-11T22:47:59.567 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.567 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap_first_committed": 1, 2020-03-11T22:47:59.567 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap_last_committed": 1, 2020-03-11T22:47:59.567 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap": { 2020-03-11T22:47:59.567 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 1, 2020-03-11T22:47:59.568 INFO:tasks.workunit.client.0.smithi145.stdout: "fsid": "e665ac71-d8d3-412f-b706-8f1d72482f80", 2020-03-11T22:47:59.568 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2020-03-11T22:45:16.173529Z", 2020-03-11T22:47:59.568 INFO:tasks.workunit.client.0.smithi145.stdout: "created": "2020-03-11T22:45:16.173529Z", 2020-03-11T22:47:59.568 INFO:tasks.workunit.client.0.smithi145.stdout: "min_mon_release": 15, 2020-03-11T22:47:59.568 INFO:tasks.workunit.client.0.smithi145.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:47:59.568 INFO:tasks.workunit.client.0.smithi145.stdout: "features": { 2020-03-11T22:47:59.569 INFO:tasks.workunit.client.0.smithi145.stdout: "persistent": [ 2020-03-11T22:47:59.569 INFO:tasks.workunit.client.0.smithi145.stdout: "kraken", 2020-03-11T22:47:59.569 INFO:tasks.workunit.client.0.smithi145.stdout: "luminous", 2020-03-11T22:47:59.569 INFO:tasks.workunit.client.0.smithi145.stdout: "mimic", 2020-03-11T22:47:59.569 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap-prune", 2020-03-11T22:47:59.570 INFO:tasks.workunit.client.0.smithi145.stdout: "nautilus", 2020-03-11T22:47:59.570 INFO:tasks.workunit.client.0.smithi145.stdout: "octopus" 2020-03-11T22:47:59.570 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:47:59.570 INFO:tasks.workunit.client.0.smithi145.stdout: "optional": [] 2020-03-11T22:47:59.570 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.570 INFO:tasks.workunit.client.0.smithi145.stdout: "mons": [ 2020-03-11T22:47:59.571 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.571 INFO:tasks.workunit.client.0.smithi145.stdout: "rank": 0, 2020-03-11T22:47:59.571 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "a", 2020-03-11T22:47:59.571 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2020-03-11T22:47:59.571 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:47:59.571 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.571 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:47:59.571 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:47:59.572 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 0 2020-03-11T22:47:59.572 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:59.572 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:59.572 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.572 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:47:59.572 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:47:59.572 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 0, 2020-03-11T22:47:59.573 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 0 2020-03-11T22:47:59.573 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:59.573 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:59.573 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.573 INFO:tasks.workunit.client.0.smithi145.stdout: "quorum": [ 2020-03-11T22:47:59.573 INFO:tasks.workunit.client.0.smithi145.stdout: 0 2020-03-11T22:47:59.573 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:47:59.573 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap": { 2020-03-11T22:47:59.574 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 47, 2020-03-11T22:47:59.574 INFO:tasks.workunit.client.0.smithi145.stdout: "fsid": "e665ac71-d8d3-412f-b706-8f1d72482f80", 2020-03-11T22:47:59.574 INFO:tasks.workunit.client.0.smithi145.stdout: "created": "2020-03-11T22:45:16.868261+0000", 2020-03-11T22:47:59.574 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2020-03-11T22:47:24.925924+0000", 2020-03-11T22:47:59.574 INFO:tasks.workunit.client.0.smithi145.stdout: "last_up_change": "2020-03-11T22:47:24.675361+0000", 2020-03-11T22:47:59.574 INFO:tasks.workunit.client.0.smithi145.stdout: "last_in_change": "2020-03-11T22:46:28.804829+0000", 2020-03-11T22:47:59.574 INFO:tasks.workunit.client.0.smithi145.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:47:59.575 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_num": 5800384, 2020-03-11T22:47:59.575 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_set": [ 2020-03-11T22:47:59.575 INFO:tasks.workunit.client.0.smithi145.stdout: "nodown", 2020-03-11T22:47:59.575 INFO:tasks.workunit.client.0.smithi145.stdout: "noin", 2020-03-11T22:47:59.575 INFO:tasks.workunit.client.0.smithi145.stdout: "noout", 2020-03-11T22:47:59.575 INFO:tasks.workunit.client.0.smithi145.stdout: "pglog_hardlimit", 2020-03-11T22:47:59.575 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snapdirs", 2020-03-11T22:47:59.575 INFO:tasks.workunit.client.0.smithi145.stdout: "recovery_deletes", 2020-03-11T22:47:59.576 INFO:tasks.workunit.client.0.smithi145.stdout: "sortbitwise" 2020-03-11T22:47:59.576 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:47:59.576 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_version": 8, 2020-03-11T22:47:59.576 INFO:tasks.workunit.client.0.smithi145.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:47:59.576 INFO:tasks.workunit.client.0.smithi145.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:47:59.576 INFO:tasks.workunit.client.0.smithi145.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:47:59.576 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_snapshot": "", 2020-03-11T22:47:59.577 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_max": 1, 2020-03-11T22:47:59.577 INFO:tasks.workunit.client.0.smithi145.stdout: "max_osd": 3, 2020-03-11T22:47:59.577 INFO:tasks.workunit.client.0.smithi145.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:47:59.577 INFO:tasks.workunit.client.0.smithi145.stdout: "min_compat_client": "jewel", 2020-03-11T22:47:59.577 INFO:tasks.workunit.client.0.smithi145.stdout: "require_osd_release": "octopus", 2020-03-11T22:47:59.577 INFO:tasks.workunit.client.0.smithi145.stdout: "pools": [ 2020-03-11T22:47:59.577 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.578 INFO:tasks.workunit.client.0.smithi145.stdout: "pool": 1, 2020-03-11T22:47:59.578 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_name": "test", 2020-03-11T22:47:59.578 INFO:tasks.workunit.client.0.smithi145.stdout: "create_time": "2020-03-11T22:45:40.413891+0000", 2020-03-11T22:47:59.578 INFO:tasks.workunit.client.0.smithi145.stdout: "flags": 16389, 2020-03-11T22:47:59.578 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2020-03-11T22:47:59.578 INFO:tasks.workunit.client.0.smithi145.stdout: "type": 3, 2020-03-11T22:47:59.579 INFO:tasks.workunit.client.0.smithi145.stdout: "size": 3, 2020-03-11T22:47:59.579 INFO:tasks.workunit.client.0.smithi145.stdout: "min_size": 2, 2020-03-11T22:47:59.579 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_rule": 1, 2020-03-11T22:47:59.579 INFO:tasks.workunit.client.0.smithi145.stdout: "object_hash": 2, 2020-03-11T22:47:59.579 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_autoscale_mode": "off", 2020-03-11T22:47:59.580 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num": 1, 2020-03-11T22:47:59.580 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_placement_num": 1, 2020-03-11T22:47:59.580 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_placement_num_target": 1, 2020-03-11T22:47:59.580 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num_target": 1, 2020-03-11T22:47:59.580 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num_pending": 1, 2020-03-11T22:47:59.580 INFO:tasks.workunit.client.0.smithi145.stdout: "last_pg_merge_meta": { 2020-03-11T22:47:59.580 INFO:tasks.workunit.client.0.smithi145.stdout: "source_pgid": "0.0", 2020-03-11T22:47:59.581 INFO:tasks.workunit.client.0.smithi145.stdout: "ready_epoch": 0, 2020-03-11T22:47:59.581 INFO:tasks.workunit.client.0.smithi145.stdout: "last_epoch_started": 0, 2020-03-11T22:47:59.581 INFO:tasks.workunit.client.0.smithi145.stdout: "last_epoch_clean": 0, 2020-03-11T22:47:59.581 INFO:tasks.workunit.client.0.smithi145.stdout: "source_version": "0'0", 2020-03-11T22:47:59.581 INFO:tasks.workunit.client.0.smithi145.stdout: "target_version": "0'0" 2020-03-11T22:47:59.581 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.581 INFO:tasks.workunit.client.0.smithi145.stdout: "last_change": "29", 2020-03-11T22:47:59.581 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend": "0", 2020-03-11T22:47:59.582 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:47:59.582 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend_preluminous": "0", 2020-03-11T22:47:59.582 INFO:tasks.workunit.client.0.smithi145.stdout: "auid": 0, 2020-03-11T22:47:59.582 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_mode": "pool", 2020-03-11T22:47:59.582 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_seq": 1, 2020-03-11T22:47:59.582 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_epoch": 29, 2020-03-11T22:47:59.582 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_snaps": [ 2020-03-11T22:47:59.583 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.583 INFO:tasks.workunit.client.0.smithi145.stdout: "snapid": 1, 2020-03-11T22:47:59.583 INFO:tasks.workunit.client.0.smithi145.stdout: "stamp": "2020-03-11T22:46:10.156100+0000", 2020-03-11T22:47:59.583 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "snap1" 2020-03-11T22:47:59.583 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:59.583 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:47:59.583 INFO:tasks.workunit.client.0.smithi145.stdout: "removed_snaps": "[]", 2020-03-11T22:47:59.583 INFO:tasks.workunit.client.0.smithi145.stdout: "quota_max_bytes": 0, 2020-03-11T22:47:59.584 INFO:tasks.workunit.client.0.smithi145.stdout: "quota_max_objects": 0, 2020-03-11T22:47:59.584 INFO:tasks.workunit.client.0.smithi145.stdout: "tiers": [], 2020-03-11T22:47:59.584 INFO:tasks.workunit.client.0.smithi145.stdout: "tier_of": -1, 2020-03-11T22:47:59.584 INFO:tasks.workunit.client.0.smithi145.stdout: "read_tier": -1, 2020-03-11T22:47:59.584 INFO:tasks.workunit.client.0.smithi145.stdout: "write_tier": -1, 2020-03-11T22:47:59.585 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_mode": "none", 2020-03-11T22:47:59.585 INFO:tasks.workunit.client.0.smithi145.stdout: "target_max_bytes": 0, 2020-03-11T22:47:59.585 INFO:tasks.workunit.client.0.smithi145.stdout: "target_max_objects": 0, 2020-03-11T22:47:59.585 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:47:59.585 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:47:59.585 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:47:59.585 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_min_flush_age": 0, 2020-03-11T22:47:59.586 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_min_evict_age": 0, 2020-03-11T22:47:59.586 INFO:tasks.workunit.client.0.smithi145.stdout: "erasure_code_profile": "myprofile", 2020-03-11T22:47:59.586 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_params": { 2020-03-11T22:47:59.586 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "none" 2020-03-11T22:47:59.586 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.586 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_period": 0, 2020-03-11T22:47:59.586 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_count": 0, 2020-03-11T22:47:59.587 INFO:tasks.workunit.client.0.smithi145.stdout: "use_gmt_hitset": true, 2020-03-11T22:47:59.587 INFO:tasks.workunit.client.0.smithi145.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:47:59.587 INFO:tasks.workunit.client.0.smithi145.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:47:59.587 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:47:59.587 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:47:59.587 INFO:tasks.workunit.client.0.smithi145.stdout: "grade_table": [], 2020-03-11T22:47:59.587 INFO:tasks.workunit.client.0.smithi145.stdout: "stripe_width": 8192, 2020-03-11T22:47:59.588 INFO:tasks.workunit.client.0.smithi145.stdout: "expected_num_objects": 0, 2020-03-11T22:47:59.588 INFO:tasks.workunit.client.0.smithi145.stdout: "fast_read": false, 2020-03-11T22:47:59.588 INFO:tasks.workunit.client.0.smithi145.stdout: "options": {}, 2020-03-11T22:47:59.588 INFO:tasks.workunit.client.0.smithi145.stdout: "application_metadata": {} 2020-03-11T22:47:59.588 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:59.589 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:47:59.589 INFO:tasks.workunit.client.0.smithi145.stdout: "osds": [ 2020-03-11T22:47:59.589 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.589 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2020-03-11T22:47:59.589 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "04216e1d-6638-4e33-8cad-1ea8a9c30138", 2020-03-11T22:47:59.589 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 1, 2020-03-11T22:47:59.590 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2020-03-11T22:47:59.590 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2020-03-11T22:47:59.590 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2020-03-11T22:47:59.590 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 35, 2020-03-11T22:47:59.590 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 41, 2020-03-11T22:47:59.590 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 43, 2020-03-11T22:47:59.591 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 46, 2020-03-11T22:47:59.591 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 42, 2020-03-11T22:47:59.591 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2020-03-11T22:47:59.591 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2020-03-11T22:47:59.591 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:47:59.591 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.591 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:47:59.591 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:47:59.592 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 50394 2020-03-11T22:47:59.592 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.592 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.592 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:47:59.592 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:47:59.592 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 50394 2020-03-11T22:47:59.592 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:59.593 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:59.593 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.593 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2020-03-11T22:47:59.593 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:47:59.593 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.593 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:47:59.593 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:47:59.593 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 50394 2020-03-11T22:47:59.594 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.594 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.594 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:47:59.595 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:47:59.595 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 50394 2020-03-11T22:47:59.595 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:59.595 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:59.595 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.596 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2020-03-11T22:47:59.596 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:47:59.596 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.596 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:47:59.596 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:47:59.597 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 50394 2020-03-11T22:47:59.597 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.597 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.597 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:47:59.597 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:47:59.597 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 50394 2020-03-11T22:47:59.597 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:59.598 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:59.598 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.598 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2020-03-11T22:47:59.598 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:47:59.598 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.598 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:47:59.598 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:47:59.599 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 50394 2020-03-11T22:47:59.599 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.599 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.599 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:47:59.599 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:47:59.599 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 50394 2020-03-11T22:47:59.599 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:59.600 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:59.600 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.600 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6803/50394", 2020-03-11T22:47:59.600 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6805/50394", 2020-03-11T22:47:59.600 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6809/50394", 2020-03-11T22:47:59.600 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6807/50394", 2020-03-11T22:47:59.600 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2020-03-11T22:47:59.601 INFO:tasks.workunit.client.0.smithi145.stdout: "exists", 2020-03-11T22:47:59.601 INFO:tasks.workunit.client.0.smithi145.stdout: "up" 2020-03-11T22:47:59.601 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:59.601 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.601 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.601 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2020-03-11T22:47:59.601 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "bc68bb65-33a0-4802-9e9b-c0869e7c2f4b", 2020-03-11T22:47:59.602 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 0, 2020-03-11T22:47:59.602 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2020-03-11T22:47:59.602 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2020-03-11T22:47:59.602 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2020-03-11T22:47:59.602 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 0, 2020-03-11T22:47:59.602 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 0, 2020-03-11T22:47:59.602 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 10, 2020-03-11T22:47:59.602 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 24, 2020-03-11T22:47:59.603 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 31, 2020-03-11T22:47:59.603 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2020-03-11T22:47:59.603 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2020-03-11T22:47:59.603 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:47:59.603 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.603 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:47:59.603 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:47:59.604 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 43891 2020-03-11T22:47:59.604 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.604 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.604 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:47:59.604 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:47:59.604 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 43891 2020-03-11T22:47:59.604 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:59.605 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:59.605 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.605 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2020-03-11T22:47:59.605 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:47:59.605 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.605 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:47:59.605 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:47:59.606 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 43891 2020-03-11T22:47:59.606 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.606 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.606 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:47:59.606 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:47:59.606 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 43891 2020-03-11T22:47:59.606 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:59.607 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:59.607 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.607 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2020-03-11T22:47:59.607 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:47:59.607 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.607 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:47:59.607 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:47:59.608 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 43891 2020-03-11T22:47:59.608 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.608 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.608 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:47:59.609 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:47:59.609 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 43891 2020-03-11T22:47:59.609 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:59.609 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:59.609 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.609 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2020-03-11T22:47:59.609 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:47:59.610 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.610 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:47:59.610 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:47:59.610 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 43891 2020-03-11T22:47:59.610 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.610 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.611 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:47:59.611 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:47:59.611 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 43891 2020-03-11T22:47:59.612 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:59.612 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:59.612 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.612 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6811/43891", 2020-03-11T22:47:59.612 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6813/43891", 2020-03-11T22:47:59.612 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6817/43891", 2020-03-11T22:47:59.612 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6815/43891", 2020-03-11T22:47:59.612 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2020-03-11T22:47:59.613 INFO:tasks.workunit.client.0.smithi145.stdout: "exists" 2020-03-11T22:47:59.613 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:59.613 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.614 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.614 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2020-03-11T22:47:59.614 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "1df352e0-156e-4ee7-a96d-63bceae4b98b", 2020-03-11T22:47:59.614 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 1, 2020-03-11T22:47:59.614 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2020-03-11T22:47:59.614 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2020-03-11T22:47:59.614 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2020-03-11T22:47:59.615 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 36, 2020-03-11T22:47:59.615 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 41, 2020-03-11T22:47:59.615 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 46, 2020-03-11T22:47:59.615 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 0, 2020-03-11T22:47:59.615 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 45, 2020-03-11T22:47:59.615 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2020-03-11T22:47:59.616 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2020-03-11T22:47:59.616 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:47:59.616 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.616 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:47:59.616 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:47:59.616 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 50626 2020-03-11T22:47:59.616 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.616 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.617 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:47:59.617 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:47:59.617 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 50626 2020-03-11T22:47:59.617 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:59.617 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:59.617 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.617 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2020-03-11T22:47:59.618 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:47:59.618 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.618 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:47:59.618 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:47:59.618 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 50626 2020-03-11T22:47:59.618 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.618 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.619 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:47:59.619 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:47:59.619 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 50626 2020-03-11T22:47:59.619 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:59.619 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:59.619 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.619 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2020-03-11T22:47:59.620 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:47:59.620 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.620 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:47:59.620 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:47:59.620 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 50626 2020-03-11T22:47:59.620 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.620 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.621 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:47:59.621 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:47:59.621 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 50626 2020-03-11T22:47:59.621 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:59.621 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:59.621 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.622 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2020-03-11T22:47:59.622 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2020-03-11T22:47:59.622 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.622 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2020-03-11T22:47:59.622 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:47:59.622 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 50626 2020-03-11T22:47:59.622 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.623 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.623 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2020-03-11T22:47:59.623 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:47:59.623 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 50626 2020-03-11T22:47:59.623 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:59.623 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:59.623 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.624 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6811/50626", 2020-03-11T22:47:59.624 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6813/50626", 2020-03-11T22:47:59.624 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6817/50626", 2020-03-11T22:47:59.624 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6815/50626", 2020-03-11T22:47:59.624 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2020-03-11T22:47:59.624 INFO:tasks.workunit.client.0.smithi145.stdout: "exists", 2020-03-11T22:47:59.624 INFO:tasks.workunit.client.0.smithi145.stdout: "up" 2020-03-11T22:47:59.625 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:59.625 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:59.625 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:47:59.625 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_xinfo": [ 2020-03-11T22:47:59.625 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.625 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2020-03-11T22:47:59.626 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "2020-03-11T22:47:22.386775+0000", 2020-03-11T22:47:59.626 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2020-03-11T22:47:59.626 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2020-03-11T22:47:59.626 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138292836696063, 2020-03-11T22:47:59.626 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2020-03-11T22:47:59.627 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2020-03-11T22:45:23.858878+0000", 2020-03-11T22:47:59.627 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2020-03-11T22:47:59.627 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.627 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.628 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2020-03-11T22:47:59.628 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "2020-03-11T22:46:21.551717+0000", 2020-03-11T22:47:59.628 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2020-03-11T22:47:59.628 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2020-03-11T22:47:59.628 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138292836696063, 2020-03-11T22:47:59.628 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2020-03-11T22:47:59.629 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2020-03-11T22:45:30.069381+0000", 2020-03-11T22:47:59.629 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2020-03-11T22:47:59.629 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.629 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.629 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2020-03-11T22:47:59.629 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "2020-03-11T22:47:24.500023+0000", 2020-03-11T22:47:59.630 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2020-03-11T22:47:59.630 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2020-03-11T22:47:59.630 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138292836696063, 2020-03-11T22:47:59.630 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2020-03-11T22:47:59.630 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2020-03-11T22:45:37.532782+0000", 2020-03-11T22:47:59.630 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2020-03-11T22:47:59.630 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:59.631 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:47:59.631 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_upmap": [], 2020-03-11T22:47:59.631 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_upmap_items": [], 2020-03-11T22:47:59.631 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_temp": [], 2020-03-11T22:47:59.631 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_temp": [], 2020-03-11T22:47:59.631 INFO:tasks.workunit.client.0.smithi145.stdout: "blacklist": {}, 2020-03-11T22:47:59.631 INFO:tasks.workunit.client.0.smithi145.stdout: "erasure_code_profiles": { 2020-03-11T22:47:59.631 INFO:tasks.workunit.client.0.smithi145.stdout: "default": { 2020-03-11T22:47:59.632 INFO:tasks.workunit.client.0.smithi145.stdout: "k": "2", 2020-03-11T22:47:59.632 INFO:tasks.workunit.client.0.smithi145.stdout: "m": "2", 2020-03-11T22:47:59.632 INFO:tasks.workunit.client.0.smithi145.stdout: "plugin": "jerasure", 2020-03-11T22:47:59.632 INFO:tasks.workunit.client.0.smithi145.stdout: "technique": "reed_sol_van" 2020-03-11T22:47:59.632 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.632 INFO:tasks.workunit.client.0.smithi145.stdout: "myprofile": { 2020-03-11T22:47:59.632 INFO:tasks.workunit.client.0.smithi145.stdout: "crush-device-class": "", 2020-03-11T22:47:59.633 INFO:tasks.workunit.client.0.smithi145.stdout: "crush-failure-domain": "osd", 2020-03-11T22:47:59.633 INFO:tasks.workunit.client.0.smithi145.stdout: "crush-root": "default", 2020-03-11T22:47:59.633 INFO:tasks.workunit.client.0.smithi145.stdout: "jerasure-per-chunk-alignment": "false", 2020-03-11T22:47:59.633 INFO:tasks.workunit.client.0.smithi145.stdout: "k": "2", 2020-03-11T22:47:59.633 INFO:tasks.workunit.client.0.smithi145.stdout: "m": "1", 2020-03-11T22:47:59.633 INFO:tasks.workunit.client.0.smithi145.stdout: "plugin": "jerasure", 2020-03-11T22:47:59.634 INFO:tasks.workunit.client.0.smithi145.stdout: "technique": "reed_sol_van", 2020-03-11T22:47:59.634 INFO:tasks.workunit.client.0.smithi145.stdout: "w": "8" 2020-03-11T22:47:59.634 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:59.634 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.634 INFO:tasks.workunit.client.0.smithi145.stdout: "removed_snaps_queue": [], 2020-03-11T22:47:59.634 INFO:tasks.workunit.client.0.smithi145.stdout: "new_removed_snaps": [], 2020-03-11T22:47:59.635 INFO:tasks.workunit.client.0.smithi145.stdout: "new_purged_snaps": [], 2020-03-11T22:47:59.635 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_node_flags": {}, 2020-03-11T22:47:59.635 INFO:tasks.workunit.client.0.smithi145.stdout: "device_class_flags": {} 2020-03-11T22:47:59.635 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.635 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_metadata": [ 2020-03-11T22:47:59.635 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.635 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2020-03-11T22:47:59.636 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2020-03-11T22:47:59.636 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6804/50394,v1:127.0.0.1:6805/50394]", 2020-03-11T22:47:59.636 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2020-03-11T22:47:59.636 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2020-03-11T22:47:59.636 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:47:59.636 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:47:59.636 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:47:59.637 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:47:59.637 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:47:59.637 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:47:59.637 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:47:59.637 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:47:59.637 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:47:59.637 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:47:59.638 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:47:59.638 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:47:59.638 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "octopus", 2020-03-11T22:47:59.638 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:47:59.638 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:47:59.638 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:47:59.638 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2020-03-11T22:47:59.639 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XLDP", 2020-03-11T22:47:59.639 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:47:59.639 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2020-03-11T22:47:59.639 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2020-03-11T22:47:59.639 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:47:59.639 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2020-03-11T22:47:59.639 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6802/50394,v1:127.0.0.1:6803/50394]", 2020-03-11T22:47:59.640 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2020-03-11T22:47:59.640 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/50394,v1:127.0.0.1:6809/50394]", 2020-03-11T22:47:59.640 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/50394,v1:127.0.0.1:6807/50394]", 2020-03-11T22:47:59.640 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2020-03-11T22:47:59.640 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2020-03-11T22:47:59.640 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:47:59.640 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:47:59.641 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2020-03-11T22:47:59.641 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32655820", 2020-03-11T22:47:59.641 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:47:59.641 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:47:59.641 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2020-03-11T22:47:59.641 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:47:59.641 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:47:59.642 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2020-03-11T22:47:59.642 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.642 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.642 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2020-03-11T22:47:59.642 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2020-03-11T22:47:59.642 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6812/43891,v1:127.0.0.1:6813/43891]", 2020-03-11T22:47:59.642 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2020-03-11T22:47:59.643 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2020-03-11T22:47:59.643 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:47:59.643 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:47:59.643 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:47:59.643 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:47:59.643 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:47:59.643 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:47:59.643 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:47:59.644 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:47:59.644 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:47:59.644 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:47:59.644 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:47:59.645 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:47:59.645 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "octopus", 2020-03-11T22:47:59.645 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:47:59.645 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:47:59.645 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:47:59.646 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2020-03-11T22:47:59.646 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XLDP", 2020-03-11T22:47:59.646 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:47:59.646 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2020-03-11T22:47:59.646 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2020-03-11T22:47:59.646 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:47:59.646 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2020-03-11T22:47:59.647 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6810/43891,v1:127.0.0.1:6811/43891]", 2020-03-11T22:47:59.647 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2020-03-11T22:47:59.647 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/43891,v1:127.0.0.1:6817/43891]", 2020-03-11T22:47:59.647 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/43891,v1:127.0.0.1:6815/43891]", 2020-03-11T22:47:59.647 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2020-03-11T22:47:59.647 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2020-03-11T22:47:59.647 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:47:59.648 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:47:59.648 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2020-03-11T22:47:59.648 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32655820", 2020-03-11T22:47:59.648 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:47:59.648 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:47:59.648 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2020-03-11T22:47:59.648 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:47:59.648 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:47:59.649 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2020-03-11T22:47:59.649 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.649 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.649 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2020-03-11T22:47:59.649 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2020-03-11T22:47:59.649 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6812/50626,v1:127.0.0.1:6813/50626]", 2020-03-11T22:47:59.650 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2020-03-11T22:47:59.650 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2020-03-11T22:47:59.650 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:47:59.650 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:47:59.650 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:47:59.650 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:47:59.650 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:47:59.650 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:47:59.651 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:47:59.651 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:47:59.651 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:47:59.651 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:47:59.651 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:47:59.651 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:47:59.651 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "octopus", 2020-03-11T22:47:59.652 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:47:59.652 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:47:59.652 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:47:59.652 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2020-03-11T22:47:59.652 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XLDP", 2020-03-11T22:47:59.652 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:47:59.652 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2020-03-11T22:47:59.653 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2020-03-11T22:47:59.653 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:47:59.653 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2020-03-11T22:47:59.653 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6810/50626,v1:127.0.0.1:6811/50626]", 2020-03-11T22:47:59.653 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2020-03-11T22:47:59.653 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/50626,v1:127.0.0.1:6817/50626]", 2020-03-11T22:47:59.653 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/50626,v1:127.0.0.1:6815/50626]", 2020-03-11T22:47:59.653 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2020-03-11T22:47:59.654 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2020-03-11T22:47:59.654 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:47:59.654 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:47:59.654 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2020-03-11T22:47:59.654 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32655820", 2020-03-11T22:47:59.654 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:47:59.654 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:47:59.655 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2020-03-11T22:47:59.655 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:47:59.655 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:47:59.655 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2020-03-11T22:47:59.655 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:59.655 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:47:59.655 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap_first_committed": 1, 2020-03-11T22:47:59.656 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap_last_committed": 47, 2020-03-11T22:47:59.656 INFO:tasks.workunit.client.0.smithi145.stdout: "crushmap": { 2020-03-11T22:47:59.656 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": [ 2020-03-11T22:47:59.656 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.656 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2020-03-11T22:47:59.656 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.0", 2020-03-11T22:47:59.656 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2020-03-11T22:47:59.657 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.657 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.657 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2020-03-11T22:47:59.657 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.1", 2020-03-11T22:47:59.657 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2020-03-11T22:47:59.657 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.657 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.658 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2020-03-11T22:47:59.658 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.2", 2020-03-11T22:47:59.658 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2020-03-11T22:47:59.658 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:59.658 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:47:59.658 INFO:tasks.workunit.client.0.smithi145.stdout: "types": [ 2020-03-11T22:47:59.658 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.659 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 0, 2020-03-11T22:47:59.659 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd" 2020-03-11T22:47:59.659 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.659 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.659 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2020-03-11T22:47:59.659 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "host" 2020-03-11T22:47:59.659 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.660 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.660 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 2, 2020-03-11T22:47:59.660 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "chassis" 2020-03-11T22:47:59.660 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.660 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.660 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 3, 2020-03-11T22:47:59.660 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "rack" 2020-03-11T22:47:59.660 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.661 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.661 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 4, 2020-03-11T22:47:59.661 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "row" 2020-03-11T22:47:59.661 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.661 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.661 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 5, 2020-03-11T22:47:59.661 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "pdu" 2020-03-11T22:47:59.662 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.662 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.662 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 6, 2020-03-11T22:47:59.662 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "pod" 2020-03-11T22:47:59.662 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.662 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.662 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 7, 2020-03-11T22:47:59.663 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "room" 2020-03-11T22:47:59.663 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.663 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.663 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 8, 2020-03-11T22:47:59.663 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "datacenter" 2020-03-11T22:47:59.663 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.663 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.664 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 9, 2020-03-11T22:47:59.664 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "zone" 2020-03-11T22:47:59.664 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.664 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.664 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 10, 2020-03-11T22:47:59.664 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "region" 2020-03-11T22:47:59.664 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.665 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.665 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2020-03-11T22:47:59.665 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "root" 2020-03-11T22:47:59.665 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:59.665 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:47:59.665 INFO:tasks.workunit.client.0.smithi145.stdout: "buckets": [ 2020-03-11T22:47:59.666 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.666 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -1, 2020-03-11T22:47:59.666 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "default", 2020-03-11T22:47:59.666 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2020-03-11T22:47:59.666 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "root", 2020-03-11T22:47:59.666 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2020-03-11T22:47:59.666 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2020-03-11T22:47:59.667 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2020-03-11T22:47:59.667 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:47:59.667 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.667 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -3, 2020-03-11T22:47:59.667 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2020-03-11T22:47:59.667 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2020-03-11T22:47:59.668 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:59.668 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:59.668 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.668 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.668 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -2, 2020-03-11T22:47:59.668 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "default~hdd", 2020-03-11T22:47:59.668 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2020-03-11T22:47:59.669 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "root", 2020-03-11T22:47:59.669 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2020-03-11T22:47:59.669 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2020-03-11T22:47:59.669 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2020-03-11T22:47:59.669 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:47:59.669 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.669 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -4, 2020-03-11T22:47:59.670 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2020-03-11T22:47:59.670 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2020-03-11T22:47:59.670 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:59.670 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:59.670 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.670 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.670 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -3, 2020-03-11T22:47:59.670 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "smithi145", 2020-03-11T22:47:59.671 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2020-03-11T22:47:59.671 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "host", 2020-03-11T22:47:59.671 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2020-03-11T22:47:59.671 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2020-03-11T22:47:59.671 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2020-03-11T22:47:59.671 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:47:59.671 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.672 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2020-03-11T22:47:59.672 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2020-03-11T22:47:59.672 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2020-03-11T22:47:59.672 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.672 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.672 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2020-03-11T22:47:59.672 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2020-03-11T22:47:59.673 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 1 2020-03-11T22:47:59.673 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.673 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.673 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2020-03-11T22:47:59.673 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2020-03-11T22:47:59.673 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 2 2020-03-11T22:47:59.674 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:59.674 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:59.674 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.674 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.674 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -4, 2020-03-11T22:47:59.675 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "smithi145~hdd", 2020-03-11T22:47:59.675 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2020-03-11T22:47:59.675 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "host", 2020-03-11T22:47:59.675 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2020-03-11T22:47:59.675 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2020-03-11T22:47:59.675 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2020-03-11T22:47:59.675 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:47:59.676 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.676 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2020-03-11T22:47:59.676 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2020-03-11T22:47:59.676 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2020-03-11T22:47:59.676 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.676 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.676 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2020-03-11T22:47:59.677 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2020-03-11T22:47:59.677 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 1 2020-03-11T22:47:59.677 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.677 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.677 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2020-03-11T22:47:59.677 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2020-03-11T22:47:59.677 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 2 2020-03-11T22:47:59.678 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:59.678 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:59.678 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:59.678 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:47:59.678 INFO:tasks.workunit.client.0.smithi145.stdout: "rules": [ 2020-03-11T22:47:59.678 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.678 INFO:tasks.workunit.client.0.smithi145.stdout: "rule_id": 0, 2020-03-11T22:47:59.679 INFO:tasks.workunit.client.0.smithi145.stdout: "rule_name": "replicated_rule", 2020-03-11T22:47:59.679 INFO:tasks.workunit.client.0.smithi145.stdout: "ruleset": 0, 2020-03-11T22:47:59.679 INFO:tasks.workunit.client.0.smithi145.stdout: "type": 1, 2020-03-11T22:47:59.679 INFO:tasks.workunit.client.0.smithi145.stdout: "min_size": 1, 2020-03-11T22:47:59.679 INFO:tasks.workunit.client.0.smithi145.stdout: "max_size": 10, 2020-03-11T22:47:59.679 INFO:tasks.workunit.client.0.smithi145.stdout: "steps": [ 2020-03-11T22:47:59.679 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.680 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "take", 2020-03-11T22:47:59.680 INFO:tasks.workunit.client.0.smithi145.stdout: "item": -1, 2020-03-11T22:47:59.680 INFO:tasks.workunit.client.0.smithi145.stdout: "item_name": "default" 2020-03-11T22:47:59.680 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.680 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.680 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "choose_firstn", 2020-03-11T22:47:59.680 INFO:tasks.workunit.client.0.smithi145.stdout: "num": 0, 2020-03-11T22:47:59.681 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "osd" 2020-03-11T22:47:59.681 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.681 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.681 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "emit" 2020-03-11T22:47:59.681 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:59.681 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:59.681 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.681 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.682 INFO:tasks.workunit.client.0.smithi145.stdout: "rule_id": 1, 2020-03-11T22:47:59.682 INFO:tasks.workunit.client.0.smithi145.stdout: "rule_name": "test", 2020-03-11T22:47:59.682 INFO:tasks.workunit.client.0.smithi145.stdout: "ruleset": 1, 2020-03-11T22:47:59.682 INFO:tasks.workunit.client.0.smithi145.stdout: "type": 3, 2020-03-11T22:47:59.682 INFO:tasks.workunit.client.0.smithi145.stdout: "min_size": 3, 2020-03-11T22:47:59.682 INFO:tasks.workunit.client.0.smithi145.stdout: "max_size": 3, 2020-03-11T22:47:59.682 INFO:tasks.workunit.client.0.smithi145.stdout: "steps": [ 2020-03-11T22:47:59.683 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.683 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "set_chooseleaf_tries", 2020-03-11T22:47:59.683 INFO:tasks.workunit.client.0.smithi145.stdout: "num": 5 2020-03-11T22:47:59.683 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.683 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.683 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "set_choose_tries", 2020-03-11T22:47:59.684 INFO:tasks.workunit.client.0.smithi145.stdout: "num": 100 2020-03-11T22:47:59.684 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.684 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.684 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "take", 2020-03-11T22:47:59.684 INFO:tasks.workunit.client.0.smithi145.stdout: "item": -1, 2020-03-11T22:47:59.685 INFO:tasks.workunit.client.0.smithi145.stdout: "item_name": "default" 2020-03-11T22:47:59.685 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.685 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.685 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "choose_indep", 2020-03-11T22:47:59.685 INFO:tasks.workunit.client.0.smithi145.stdout: "num": 0, 2020-03-11T22:47:59.685 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "osd" 2020-03-11T22:47:59.686 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.686 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.686 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "emit" 2020-03-11T22:47:59.686 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:59.686 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:47:59.686 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:59.686 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:47:59.687 INFO:tasks.workunit.client.0.smithi145.stdout: "tunables": { 2020-03-11T22:47:59.687 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_local_tries": 0, 2020-03-11T22:47:59.687 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:47:59.687 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_total_tries": 50, 2020-03-11T22:47:59.687 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:47:59.687 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:47:59.687 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_stable": 1, 2020-03-11T22:47:59.688 INFO:tasks.workunit.client.0.smithi145.stdout: "straw_calc_version": 1, 2020-03-11T22:47:59.688 INFO:tasks.workunit.client.0.smithi145.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:47:59.688 INFO:tasks.workunit.client.0.smithi145.stdout: "profile": "jewel", 2020-03-11T22:47:59.688 INFO:tasks.workunit.client.0.smithi145.stdout: "optimal_tunables": 1, 2020-03-11T22:47:59.688 INFO:tasks.workunit.client.0.smithi145.stdout: "legacy_tunables": 0, 2020-03-11T22:47:59.688 INFO:tasks.workunit.client.0.smithi145.stdout: "minimum_required_version": "jewel", 2020-03-11T22:47:59.688 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables": 1, 2020-03-11T22:47:59.688 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables2": 1, 2020-03-11T22:47:59.689 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v2_rules": 1, 2020-03-11T22:47:59.689 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables3": 1, 2020-03-11T22:47:59.689 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v3_rules": 0, 2020-03-11T22:47:59.689 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v4_buckets": 1, 2020-03-11T22:47:59.689 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables5": 1, 2020-03-11T22:47:59.689 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v5_rules": 0 2020-03-11T22:47:59.689 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.690 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_args": {} 2020-03-11T22:47:59.690 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.690 INFO:tasks.workunit.client.0.smithi145.stdout: "fsmap": { 2020-03-11T22:47:59.690 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 1, 2020-03-11T22:47:59.690 INFO:tasks.workunit.client.0.smithi145.stdout: "default_fscid": -1, 2020-03-11T22:47:59.690 INFO:tasks.workunit.client.0.smithi145.stdout: "compat": { 2020-03-11T22:47:59.690 INFO:tasks.workunit.client.0.smithi145.stdout: "compat": {}, 2020-03-11T22:47:59.691 INFO:tasks.workunit.client.0.smithi145.stdout: "ro_compat": {}, 2020-03-11T22:47:59.691 INFO:tasks.workunit.client.0.smithi145.stdout: "incompat": { 2020-03-11T22:47:59.691 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_1": "base v0.20", 2020-03-11T22:47:59.691 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:47:59.691 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:47:59.691 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:47:59.691 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:47:59.692 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:47:59.692 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_8": "no anchor table", 2020-03-11T22:47:59.692 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_9": "file layout v2", 2020-03-11T22:47:59.692 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:47:59.692 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:59.692 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.692 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_flags": { 2020-03-11T22:47:59.693 INFO:tasks.workunit.client.0.smithi145.stdout: "enable_multiple": false, 2020-03-11T22:47:59.693 INFO:tasks.workunit.client.0.smithi145.stdout: "ever_enabled_multiple": false 2020-03-11T22:47:59.693 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.693 INFO:tasks.workunit.client.0.smithi145.stdout: "standbys": [], 2020-03-11T22:47:59.693 INFO:tasks.workunit.client.0.smithi145.stdout: "filesystems": [] 2020-03-11T22:47:59.693 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.693 INFO:tasks.workunit.client.0.smithi145.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:47:59.694 INFO:tasks.workunit.client.0.smithi145.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:47:59.694 INFO:tasks.workunit.client.0.smithi145.stdout: "auth": { 2020-03-11T22:47:59.694 INFO:tasks.workunit.client.0.smithi145.stdout: "first_committed": 1, 2020-03-11T22:47:59.694 INFO:tasks.workunit.client.0.smithi145.stdout: "last_committed": 5, 2020-03-11T22:47:59.694 INFO:tasks.workunit.client.0.smithi145.stdout: "num_secrets": 3 2020-03-11T22:47:59.694 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.694 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg": 1, 2020-03-11T22:47:59.694 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_active": 1, 2020-03-11T22:47:59.695 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_unknown": 0, 2020-03-11T22:47:59.695 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osd": 3, 2020-03-11T22:47:59.695 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_sum": { 2020-03-11T22:47:59.695 INFO:tasks.workunit.client.0.smithi145.stdout: "stat_sum": { 2020-03-11T22:47:59.695 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes": 48430, 2020-03-11T22:47:59.695 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects": 101, 2020-03-11T22:47:59.695 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_clones": 1, 2020-03-11T22:47:59.696 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_copies": 303, 2020-03-11T22:47:59.696 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:47:59.696 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing": 0, 2020-03-11T22:47:59.696 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_degraded": 101, 2020-03-11T22:47:59.696 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_misplaced": 0, 2020-03-11T22:47:59.697 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_unfound": 0, 2020-03-11T22:47:59.697 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_dirty": 101, 2020-03-11T22:47:59.697 INFO:tasks.workunit.client.0.smithi145.stdout: "num_whiteouts": 0, 2020-03-11T22:47:59.697 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read": 0, 2020-03-11T22:47:59.697 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read_kb": 0, 2020-03-11T22:47:59.697 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write": 101, 2020-03-11T22:47:59.697 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write_kb": 101, 2020-03-11T22:47:59.698 INFO:tasks.workunit.client.0.smithi145.stdout: "num_scrub_errors": 0, 2020-03-11T22:47:59.698 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:47:59.698 INFO:tasks.workunit.client.0.smithi145.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:47:59.698 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_recovered": 0, 2020-03-11T22:47:59.698 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_recovered": 0, 2020-03-11T22:47:59.698 INFO:tasks.workunit.client.0.smithi145.stdout: "num_keys_recovered": 0, 2020-03-11T22:47:59.698 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_omap": 0, 2020-03-11T22:47:59.699 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:47:59.699 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:47:59.699 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush": 0, 2020-03-11T22:47:59.699 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_kb": 0, 2020-03-11T22:47:59.699 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict": 0, 2020-03-11T22:47:59.699 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_kb": 0, 2020-03-11T22:47:59.700 INFO:tasks.workunit.client.0.smithi145.stdout: "num_promote": 0, 2020-03-11T22:47:59.700 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_high": 0, 2020-03-11T22:47:59.700 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_low": 0, 2020-03-11T22:47:59.700 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_some": 0, 2020-03-11T22:47:59.700 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_full": 0, 2020-03-11T22:47:59.700 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_pinned": 0, 2020-03-11T22:47:59.700 INFO:tasks.workunit.client.0.smithi145.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:47:59.701 INFO:tasks.workunit.client.0.smithi145.stdout: "num_large_omap_objects": 0, 2020-03-11T22:47:59.701 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_manifest": 0, 2020-03-11T22:47:59.701 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_bytes": 0, 2020-03-11T22:47:59.701 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_keys": 0, 2020-03-11T22:47:59.701 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_repaired": 0 2020-03-11T22:47:59.701 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.701 INFO:tasks.workunit.client.0.smithi145.stdout: "store_stats": { 2020-03-11T22:47:59.702 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 0, 2020-03-11T22:47:59.702 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 0, 2020-03-11T22:47:59.702 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2020-03-11T22:47:59.702 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 0, 2020-03-11T22:47:59.702 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 0, 2020-03-11T22:47:59.702 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2020-03-11T22:47:59.702 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2020-03-11T22:47:59.702 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2020-03-11T22:47:59.703 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 0, 2020-03-11T22:47:59.703 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 0 2020-03-11T22:47:59.703 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.703 INFO:tasks.workunit.client.0.smithi145.stdout: "log_size": 18, 2020-03-11T22:47:59.703 INFO:tasks.workunit.client.0.smithi145.stdout: "ondisk_log_size": 18, 2020-03-11T22:47:59.703 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 3, 2020-03-11T22:47:59.703 INFO:tasks.workunit.client.0.smithi145.stdout: "acting": 3, 2020-03-11T22:47:59.704 INFO:tasks.workunit.client.0.smithi145.stdout: "num_store_stats": 0 2020-03-11T22:47:59.704 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.704 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_sum": { 2020-03-11T22:47:59.704 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 0, 2020-03-11T22:47:59.704 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 0, 2020-03-11T22:47:59.704 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pgs": 3, 2020-03-11T22:47:59.704 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osds": 3, 2020-03-11T22:47:59.705 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_osds": 3, 2020-03-11T22:47:59.705 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:47:59.705 INFO:tasks.workunit.client.0.smithi145.stdout: "kb": 314572800, 2020-03-11T22:47:59.705 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used": 3181568, 2020-03-11T22:47:59.705 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_data": 35520, 2020-03-11T22:47:59.705 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_omap": 10, 2020-03-11T22:47:59.705 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_meta": 3145717, 2020-03-11T22:47:59.706 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_avail": 311391232, 2020-03-11T22:47:59.706 INFO:tasks.workunit.client.0.smithi145.stdout: "statfs": { 2020-03-11T22:47:59.706 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 322122547200, 2020-03-11T22:47:59.706 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 318864621568, 2020-03-11T22:47:59.706 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 3221225472, 2020-03-11T22:47:59.706 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 36372480, 2020-03-11T22:47:59.706 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 1567561, 2020-03-11T22:47:59.706 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2020-03-11T22:47:59.707 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2020-03-11T22:47:59.707 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2020-03-11T22:47:59.707 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 10656, 2020-03-11T22:47:59.707 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 3221214816 2020-03-11T22:47:59.707 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.707 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_peers": [], 2020-03-11T22:47:59.707 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:47:59.708 INFO:tasks.workunit.client.0.smithi145.stdout: "num_snap_trimming": 0, 2020-03-11T22:47:59.708 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shards_repaired": 0, 2020-03-11T22:47:59.708 INFO:tasks.workunit.client.0.smithi145.stdout: "op_queue_age_hist": { 2020-03-11T22:47:59.708 INFO:tasks.workunit.client.0.smithi145.stdout: "histogram": [], 2020-03-11T22:47:59.708 INFO:tasks.workunit.client.0.smithi145.stdout: "upper_bound": 1 2020-03-11T22:47:59.708 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.708 INFO:tasks.workunit.client.0.smithi145.stdout: "perf_stat": { 2020-03-11T22:47:59.709 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ms": 0, 2020-03-11T22:47:59.709 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ms": 0, 2020-03-11T22:47:59.709 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ns": 0, 2020-03-11T22:47:59.709 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ns": 0 2020-03-11T22:47:59.709 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.709 INFO:tasks.workunit.client.0.smithi145.stdout: "alerts": [], 2020-03-11T22:47:59.710 INFO:tasks.workunit.client.0.smithi145.stdout: "network_ping_times": [] 2020-03-11T22:47:59.710 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.710 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_sum_by_class": { 2020-03-11T22:47:59.710 INFO:tasks.workunit.client.0.smithi145.stdout: "hdd": { 2020-03-11T22:47:59.710 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 0, 2020-03-11T22:47:59.710 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 0, 2020-03-11T22:47:59.710 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pgs": 3, 2020-03-11T22:47:59.710 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osds": 3, 2020-03-11T22:47:59.711 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_osds": 3, 2020-03-11T22:47:59.711 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:47:59.711 INFO:tasks.workunit.client.0.smithi145.stdout: "kb": 314572800, 2020-03-11T22:47:59.711 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used": 3181568, 2020-03-11T22:47:59.711 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_data": 35520, 2020-03-11T22:47:59.711 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_omap": 10, 2020-03-11T22:47:59.711 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_meta": 3145717, 2020-03-11T22:47:59.712 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_avail": 311391232, 2020-03-11T22:47:59.712 INFO:tasks.workunit.client.0.smithi145.stdout: "statfs": { 2020-03-11T22:47:59.712 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 322122547200, 2020-03-11T22:47:59.712 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 318864621568, 2020-03-11T22:47:59.712 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 3221225472, 2020-03-11T22:47:59.712 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 36372480, 2020-03-11T22:47:59.712 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 1567561, 2020-03-11T22:47:59.713 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2020-03-11T22:47:59.713 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2020-03-11T22:47:59.713 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2020-03-11T22:47:59.713 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 10656, 2020-03-11T22:47:59.713 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 3221214816 2020-03-11T22:47:59.713 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.713 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_peers": [], 2020-03-11T22:47:59.714 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:47:59.714 INFO:tasks.workunit.client.0.smithi145.stdout: "num_snap_trimming": 0, 2020-03-11T22:47:59.714 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shards_repaired": 0, 2020-03-11T22:47:59.714 INFO:tasks.workunit.client.0.smithi145.stdout: "op_queue_age_hist": { 2020-03-11T22:47:59.714 INFO:tasks.workunit.client.0.smithi145.stdout: "histogram": [], 2020-03-11T22:47:59.714 INFO:tasks.workunit.client.0.smithi145.stdout: "upper_bound": 1 2020-03-11T22:47:59.714 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.714 INFO:tasks.workunit.client.0.smithi145.stdout: "perf_stat": { 2020-03-11T22:47:59.715 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ms": 0, 2020-03-11T22:47:59.715 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ms": 0, 2020-03-11T22:47:59.715 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ns": 0, 2020-03-11T22:47:59.715 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ns": 0 2020-03-11T22:47:59.715 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.715 INFO:tasks.workunit.client.0.smithi145.stdout: "alerts": [], 2020-03-11T22:47:59.715 INFO:tasks.workunit.client.0.smithi145.stdout: "network_ping_times": [] 2020-03-11T22:47:59.716 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:59.716 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.716 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_stats": [ 2020-03-11T22:47:59.716 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.716 INFO:tasks.workunit.client.0.smithi145.stdout: "poolid": 1, 2020-03-11T22:47:59.716 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg": 1, 2020-03-11T22:47:59.716 INFO:tasks.workunit.client.0.smithi145.stdout: "stat_sum": { 2020-03-11T22:47:59.717 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes": 48430, 2020-03-11T22:47:59.717 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects": 101, 2020-03-11T22:47:59.717 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_clones": 1, 2020-03-11T22:47:59.717 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_copies": 303, 2020-03-11T22:47:59.717 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:47:59.717 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing": 0, 2020-03-11T22:47:59.717 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_degraded": 101, 2020-03-11T22:47:59.718 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_misplaced": 0, 2020-03-11T22:47:59.718 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_unfound": 0, 2020-03-11T22:47:59.718 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_dirty": 101, 2020-03-11T22:47:59.718 INFO:tasks.workunit.client.0.smithi145.stdout: "num_whiteouts": 0, 2020-03-11T22:47:59.718 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read": 0, 2020-03-11T22:47:59.718 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read_kb": 0, 2020-03-11T22:47:59.718 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write": 101, 2020-03-11T22:47:59.719 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write_kb": 101, 2020-03-11T22:47:59.719 INFO:tasks.workunit.client.0.smithi145.stdout: "num_scrub_errors": 0, 2020-03-11T22:47:59.719 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:47:59.719 INFO:tasks.workunit.client.0.smithi145.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:47:59.719 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_recovered": 0, 2020-03-11T22:47:59.719 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_recovered": 0, 2020-03-11T22:47:59.719 INFO:tasks.workunit.client.0.smithi145.stdout: "num_keys_recovered": 0, 2020-03-11T22:47:59.720 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_omap": 0, 2020-03-11T22:47:59.720 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:47:59.720 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:47:59.720 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush": 0, 2020-03-11T22:47:59.720 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_kb": 0, 2020-03-11T22:47:59.720 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict": 0, 2020-03-11T22:47:59.720 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_kb": 0, 2020-03-11T22:47:59.721 INFO:tasks.workunit.client.0.smithi145.stdout: "num_promote": 0, 2020-03-11T22:47:59.721 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_high": 0, 2020-03-11T22:47:59.721 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_low": 0, 2020-03-11T22:47:59.721 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_some": 0, 2020-03-11T22:47:59.721 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_full": 0, 2020-03-11T22:47:59.721 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_pinned": 0, 2020-03-11T22:47:59.721 INFO:tasks.workunit.client.0.smithi145.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:47:59.722 INFO:tasks.workunit.client.0.smithi145.stdout: "num_large_omap_objects": 0, 2020-03-11T22:47:59.722 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_manifest": 0, 2020-03-11T22:47:59.722 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_bytes": 0, 2020-03-11T22:47:59.722 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_keys": 0, 2020-03-11T22:47:59.722 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_repaired": 0 2020-03-11T22:47:59.723 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.723 INFO:tasks.workunit.client.0.smithi145.stdout: "store_stats": { 2020-03-11T22:47:59.723 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 0, 2020-03-11T22:47:59.723 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 0, 2020-03-11T22:47:59.723 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2020-03-11T22:47:59.723 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 19857408, 2020-03-11T22:47:59.723 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 1241088, 2020-03-11T22:47:59.723 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2020-03-11T22:47:59.724 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2020-03-11T22:47:59.724 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2020-03-11T22:47:59.724 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 0, 2020-03-11T22:47:59.724 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 0 2020-03-11T22:47:59.724 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.724 INFO:tasks.workunit.client.0.smithi145.stdout: "log_size": 18, 2020-03-11T22:47:59.724 INFO:tasks.workunit.client.0.smithi145.stdout: "ondisk_log_size": 18, 2020-03-11T22:47:59.725 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 3, 2020-03-11T22:47:59.725 INFO:tasks.workunit.client.0.smithi145.stdout: "acting": 3, 2020-03-11T22:47:59.725 INFO:tasks.workunit.client.0.smithi145.stdout: "num_store_stats": 3 2020-03-11T22:47:59.725 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:59.725 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:47:59.725 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_stats": [ 2020-03-11T22:47:59.725 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.726 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2020-03-11T22:47:59.726 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 184683593738 2020-03-11T22:47:59.726 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.726 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.726 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2020-03-11T22:47:59.726 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 42949672980 2020-03-11T22:47:59.727 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.727 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.727 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2020-03-11T22:47:59.727 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 197568495625 2020-03-11T22:47:59.728 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:59.728 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:47:59.728 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_by_state": [ 2020-03-11T22:47:59.728 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.728 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "active+undersized+degraded", 2020-03-11T22:47:59.729 INFO:tasks.workunit.client.0.smithi145.stdout: "num": 1 2020-03-11T22:47:59.729 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:59.729 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:47:59.729 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_by_osd": [ 2020-03-11T22:47:59.730 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.730 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2020-03-11T22:47:59.730 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 0, 2020-03-11T22:47:59.730 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 1, 2020-03-11T22:47:59.731 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:47:59.731 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.731 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.731 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2147483647, 2020-03-11T22:47:59.731 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 0, 2020-03-11T22:47:59.731 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 1, 2020-03-11T22:47:59.731 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:47:59.731 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.732 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.732 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2020-03-11T22:47:59.732 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 0, 2020-03-11T22:47:59.732 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 0, 2020-03-11T22:47:59.732 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:47:59.732 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.732 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.733 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2020-03-11T22:47:59.733 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 1, 2020-03-11T22:47:59.733 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 1, 2020-03-11T22:47:59.733 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:47:59.733 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:59.733 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:47:59.733 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snaps": [ 2020-03-11T22:47:59.734 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:47:59.734 INFO:tasks.workunit.client.0.smithi145.stdout: "pool": 1, 2020-03-11T22:47:59.734 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snaps": {} 2020-03-11T22:47:59.734 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:59.734 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:47:59.734 INFO:tasks.workunit.client.0.smithi145.stdout: "servicemap": { 2020-03-11T22:47:59.734 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 1, 2020-03-11T22:47:59.735 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2020-03-11T22:45:16.868571+0000", 2020-03-11T22:47:59.735 INFO:tasks.workunit.client.0.smithi145.stdout: "services": {} 2020-03-11T22:47:59.735 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:47:59.735 INFO:tasks.workunit.client.0.smithi145.stdout: "paxos": { 2020-03-11T22:47:59.735 INFO:tasks.workunit.client.0.smithi145.stdout: "first_committed": 1, 2020-03-11T22:47:59.736 INFO:tasks.workunit.client.0.smithi145.stdout: "last_committed": 455, 2020-03-11T22:47:59.736 INFO:tasks.workunit.client.0.smithi145.stdout: "last_pn": 0, 2020-03-11T22:47:59.736 INFO:tasks.workunit.client.0.smithi145.stdout: "accepted_pn": 0 2020-03-11T22:47:59.736 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:47:59.736 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:47:59.737 INFO:tasks.workunit.client.0.smithi145.stderr:report 1897163292 2020-03-11T22:47:59.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:47:59.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:341: TEST_divergent_ec: echo 'delay recovery' 2020-03-11T22:47:59.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:342: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:47:59.738 INFO:tasks.workunit.client.0.smithi145.stdout:delay recovery 2020-03-11T22:47:59.890 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:47:59.890 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2020-03-11T22:47:25.085076+0000 41'102 47:145 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-03-11T22:45:40.484776+0000 0'0 2020-03-11T22:45:40.484776+0000 0 2020-03-11T22:47:59.890 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:47:59.890 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:47:59.891 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:47:59.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:47:59.901 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.0 2020-03-11T22:47:59.901 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:47:59.901 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:47:59.901 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:47:59.901 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:59.901 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:47:59.901 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17402/ceph-osd.0.asok 2020-03-11T22:47:59.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2020-03-11T22:47:59.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.17402/ceph-osd.0.asok set_recovery_delay 100000 2020-03-11T22:47:59.991 INFO:tasks.workunit.client.0.smithi145.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:47:59.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:48:00.000 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.2 2020-03-11T22:48:00.000 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:48:00.000 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:48:00.000 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:48:00.000 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:00.000 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:48:00.001 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17402/ceph-osd.2.asok 2020-03-11T22:48:00.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2020-03-11T22:48:00.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.17402/ceph-osd.2.asok set_recovery_delay 100000 2020-03-11T22:48:00.091 INFO:tasks.workunit.client.0.smithi145.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:48:00.098 INFO:tasks.workunit.client.0.smithi145.stdout:revive divergent 1 2020-03-11T22:48:00.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:349: TEST_divergent_ec: echo 'revive divergent 1' 2020-03-11T22:48:00.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:350: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:48:00.419 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:48:00.419 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2020-03-11T22:47:25.085076+0000 41'102 47:145 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-03-11T22:45:40.484776+0000 0'0 2020-03-11T22:45:40.484776+0000 0 2020-03-11T22:48:00.420 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:48:00.420 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:48:00.421 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:48:00.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:351: TEST_divergent_ec: ceph osd set noup 2020-03-11T22:48:01.073 INFO:tasks.workunit.client.0.smithi145.stderr:noup is set 2020-03-11T22:48:01.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:352: TEST_divergent_ec: activate_osd td/divergent-priors 1 2020-03-11T22:48:01.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:48:01.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:48:01.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:48:01.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:48:01.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:48:01.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:48:01.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:48:01.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:48:01.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:48:01.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:48:01.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:48:01.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:48:01.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:48:01.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:48:01.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:48:01.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:48:01.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:48:01.087 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:48:01.087 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:01.087 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:48:01.088 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:48:01.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:48:01.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:48:01.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:48:01.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:48:01.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:48:01.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:48:01.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:01.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:48:01.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:48:01.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:48:01.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:48:01.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:01.090 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:48:01.090 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2020-03-11T22:48:01.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:48:01.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:48:01.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:48:01.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:48:01.111 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:48:01.107+0000 7fe69752dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:01.113 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:48:01.109+0000 7fe69752dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:01.114 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:48:01.110+0000 7fe69752dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:01.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: return 0 2020-03-11T22:48:01.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:353: TEST_divergent_ec: sleep 5 2020-03-11T22:48:01.629 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:48:01.626+0000 7fe69752dec0 -1 Falling back to public interface 2020-03-11T22:48:02.440 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:48:02.436+0000 7fe69752dec0 -1 osd.1 29 log_to_monitors {default=true} 2020-03-11T22:48:06.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:355: TEST_divergent_ec: echo 'delay recovery divergent' 2020-03-11T22:48:06.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:356: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:48:06.438 INFO:tasks.workunit.client.0.smithi145.stdout:delay recovery divergent 2020-03-11T22:48:06.768 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:48:06.768 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2020-03-11T22:47:25.085076+0000 41'102 47:146 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-03-11T22:45:40.484776+0000 0'0 2020-03-11T22:45:40.484776+0000 0 2020-03-11T22:48:06.768 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:48:06.768 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:48:06.769 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:48:06.778 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: get_asok_path osd.1 2020-03-11T22:48:06.778 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:48:06.779 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:48:06.779 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:48:06.779 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:06.779 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:48:06.779 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17402/ceph-osd.1.asok 2020-03-11T22:48:06.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: CEPH_ARGS= 2020-03-11T22:48:06.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.17402/ceph-osd.1.asok set_recovery_delay 100000 2020-03-11T22:48:06.870 INFO:tasks.workunit.client.0.smithi145.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:48:06.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:359: TEST_divergent_ec: ceph osd unset noup 2020-03-11T22:48:07.653 INFO:tasks.workunit.client.0.smithi145.stderr:noup is unset 2020-03-11T22:48:07.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:361: TEST_divergent_ec: wait_for_osd up 0 2020-03-11T22:48:07.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:48:07.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:48:07.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:48:07.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:48:07.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:07.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:48:07.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:07.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:48:07.663 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:48:08.018 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 43 up_thru 46 down_at 42 last_clean_interval [35,41) [v2:127.0.0.1:6802/50394,v1:127.0.0.1:6803/50394] [v2:127.0.0.1:6804/50394,v1:127.0.0.1:6805/50394] exists,up 04216e1d-6638-4e33-8cad-1ea8a9c30138 2020-03-11T22:48:08.018 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:48:08.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:48:08.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:48:08.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:48:08.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:362: TEST_divergent_ec: wait_for_osd up 1 2020-03-11T22:48:08.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:48:08.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:48:08.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:48:08.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:48:08.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:08.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:48:08.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:08.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:48:08.362 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 51 up_thru 51 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/52022,v1:127.0.0.1:6819/52022] [v2:127.0.0.1:6820/52022,v1:127.0.0.1:6821/52022] exists,up bc68bb65-33a0-4802-9e9b-c0869e7c2f4b 2020-03-11T22:48:08.362 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:48:08.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:48:08.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:48:08.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:48:08.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:363: TEST_divergent_ec: wait_for_osd up 2 2020-03-11T22:48:08.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:48:08.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:48:08.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:48:08.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:48:08.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:08.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:48:08.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:08.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:48:08.698 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 46 up_thru 0 down_at 45 last_clean_interval [36,41) [v2:127.0.0.1:6810/50626,v1:127.0.0.1:6811/50626] [v2:127.0.0.1:6812/50626,v1:127.0.0.1:6813/50626] exists,up 1df352e0-156e-4ee7-a96d-63bceae4b98b 2020-03-11T22:48:08.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:48:08.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:48:08.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:48:08.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:365: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:48:09.018 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:48:09.018 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2020-03-11T22:47:25.085076+0000 41'102 47:146 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-03-11T22:45:40.484776+0000 0'0 2020-03-11T22:45:40.484776+0000 0 2020-03-11T22:48:09.019 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:48:09.019 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:48:09.020 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:48:09.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:366: TEST_divergent_ec: echo 'wait for peering' 2020-03-11T22:48:09.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:367: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:48:09.028 INFO:tasks.workunit.client.0.smithi145.stdout:wait for peering 2020-03-11T22:48:09.350 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:48:09.350 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2020-03-11T22:47:25.085076+0000 41'102 47:146 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-03-11T22:45:40.484776+0000 0'0 2020-03-11T22:45:40.484776+0000 0 2020-03-11T22:48:09.350 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:48:09.350 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:48:09.351 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:48:09.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:368: TEST_divergent_ec: rados -p test put foo /etc/fstab 2020-03-11T22:48:21.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:370: TEST_divergent_ec: echo 'killing divergent 1' 2020-03-11T22:48:21.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:371: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:48:21.109 INFO:tasks.workunit.client.0.smithi145.stdout:killing divergent 1 2020-03-11T22:48:21.437 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:48:21.437 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 101 4 4 0 0 48430 0 0 12 12 active+recovering+degraded+wait 2020-03-11T22:48:07.749578+0000 41'102 52:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:45:40.484776+0000 0'0 2020-03-11T22:45:40.484776+0000 0 2020-03-11T22:48:21.437 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:48:21.437 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:48:21.439 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:48:21.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:372: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.1 2020-03-11T22:48:21.447 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:48:21.447 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:48:21.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:48:21.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:48:21.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:48:21.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:48:21.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:374: TEST_divergent_ec: echo 'reviving divergent 1' 2020-03-11T22:48:21.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:375: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:48:21.553 INFO:tasks.workunit.client.0.smithi145.stdout:reviving divergent 1 2020-03-11T22:48:21.872 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:48:21.872 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 101 4 4 0 0 48430 0 0 12 12 active+recovering+degraded+wait 2020-03-11T22:48:07.749578+0000 41'102 52:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:45:40.484776+0000 0'0 2020-03-11T22:45:40.484776+0000 0 2020-03-11T22:48:21.873 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:48:21.873 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:48:21.874 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:48:21.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:376: TEST_divergent_ec: activate_osd td/divergent-priors 1 2020-03-11T22:48:21.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:48:21.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:48:21.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:48:21.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:48:21.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:48:21.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:48:21.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:48:21.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:48:21.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:48:21.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:48:21.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:48:21.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:48:21.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:48:21.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:48:21.887 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:48:21.888 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:48:21.888 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:48:21.888 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:48:21.888 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:21.888 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:48:21.888 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:48:21.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' 2020-03-11T22:48:21.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:48:21.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:48:21.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:48:21.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:48:21.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:48:21.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:21.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:48:21.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:48:21.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:48:21.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:48:21.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=e665ac71-d8d3-412f-b706-8f1d72482f80 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17402/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:21.891 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:48:21.891 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2020-03-11T22:48:21.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:48:21.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:48:21.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:48:21.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:48:21.911 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:48:21.909+0000 7f82351ecec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:21.913 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:48:21.910+0000 7f82351ecec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:21.914 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:48:21.911+0000 7f82351ecec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:22.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-03-11T22:48:22.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:48:22.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:48:22.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:48:22.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:48:22.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:22.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:48:22.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:22.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:48:22.232 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:48:22.433 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:48:22.430+0000 7f82351ecec0 -1 Falling back to public interface 2020-03-11T22:48:22.577 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 51 up_thru 51 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/52022,v1:127.0.0.1:6819/52022] [v2:127.0.0.1:6820/52022,v1:127.0.0.1:6821/52022] exists,up bc68bb65-33a0-4802-9e9b-c0869e7c2f4b 2020-03-11T22:48:22.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:48:22.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:48:22.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:48:22.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:378: TEST_divergent_ec: sleep 20 2020-03-11T22:48:23.264 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:48:23.261+0000 7f82351ecec0 -1 osd.1 52 log_to_monitors {default=true} 2020-03-11T22:48:42.578 INFO:tasks.workunit.client.0.smithi145.stdout:allowing recovery 2020-03-11T22:48:42.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:380: TEST_divergent_ec: echo 'allowing recovery' 2020-03-11T22:48:42.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:381: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:48:42.909 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:48:42.909 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 102 0 0 0 0 48913 0 0 13 13 active+clean 2020-03-11T22:48:24.407867+0000 52'103 55:152 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:45:40.484776+0000 0'0 2020-03-11T22:45:40.484776+0000 0 2020-03-11T22:48:42.910 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:48:42.910 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:48:42.911 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:48:42.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2020-03-11T22:48:42.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.0 debug kick_recovery_wq 0 2020-03-11T22:48:43.040 INFO:tasks.workunit.client.0.smithi145.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:48:43.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2020-03-11T22:48:43.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.1 debug kick_recovery_wq 0 2020-03-11T22:48:43.168 INFO:tasks.workunit.client.0.smithi145.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:48:43.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2020-03-11T22:48:43.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.2 debug kick_recovery_wq 0 2020-03-11T22:48:43.296 INFO:tasks.workunit.client.0.smithi145.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:48:43.303 INFO:tasks.workunit.client.0.smithi145.stdout:reading divergent objects 2020-03-11T22:48:43.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:388: TEST_divergent_ec: echo 'reading divergent objects' 2020-03-11T22:48:43.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:389: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:48:43.620 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:48:43.620 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 102 0 0 0 0 48913 0 0 13 13 active+clean 2020-03-11T22:48:24.407867+0000 52'103 55:152 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:45:40.484776+0000 0'0 2020-03-11T22:45:40.484776+0000 0 2020-03-11T22:48:43.621 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:48:43.621 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:48:43.622 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:48:43.630 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: expr 2 + 2 2020-03-11T22:48:43.631 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: seq 1 4 2020-03-11T22:48:43.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:48:43.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_1 td/divergent-priors/existing 2020-03-11T22:48:43.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:48:43.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_2 td/divergent-priors/existing 2020-03-11T22:48:43.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:48:43.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_3 td/divergent-priors/existing 2020-03-11T22:48:43.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:48:43.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_4 td/divergent-priors/existing 2020-03-11T22:48:43.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:394: TEST_divergent_ec: rm -f td/divergent-priors/existing 2020-03-11T22:48:43.764 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2020-03-11T22:48:43.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:48:43.768 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:07.689+0000 7fe6756e8700 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:1 entries: 29'103 (28'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2020-03-11T22:46:02.424021+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:48:43.769 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:07.689+0000 7fe6756e8700 20 _merge_object_divergent_entries: keeping 29'103 (28'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2020-03-11T22:46:02.424021+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:48:43.769 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:07.689+0000 7fe6756e8700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:48:43.769 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:07.689+0000 7fe6756e8700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version: 28'99 first_divergent_update: 29'103 last_divergent_update: 29'103 2020-03-11T22:48:43.769 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:07.689+0000 7fe6756e8700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 has no more recent entries in log 2020-03-11T22:48:43.769 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:07.689+0000 7fe6756e8700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version or op type indicates creation, deleting 2020-03-11T22:48:43.769 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:07.689+0000 7fe6756e8700 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:head entries: 29'104 (28'99) modify 1:7a80c0d5:::existing_99:head by client.4563.0:1 2020-03-11T22:46:10.308803+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:48:43.770 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:07.689+0000 7fe6756e8700 20 _merge_object_divergent_entries: keeping 29'104 (28'99) modify 1:7a80c0d5:::existing_99:head by client.4563.0:1 2020-03-11T22:46:10.308803+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:48:43.770 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:07.689+0000 7fe6756e8700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:48:43.770 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:07.689+0000 7fe6756e8700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head prior_version: 28'99 first_divergent_update: 29'104 last_divergent_update: 29'104 2020-03-11T22:48:43.770 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:07.689+0000 7fe6756e8700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head has no more recent entries in log 2020-03-11T22:48:43.770 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:07.689+0000 7fe6756e8700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:48:43.770 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:07.689+0000 7fe6756e8700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head olog_can_rollback_to: 29'104 2020-03-11T22:48:43.770 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:07.689+0000 7fe6756e8700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot rollback 29'104 (28'99) modify 1:7a80c0d5:::existing_99:head by client.4563.0:1 2020-03-11T22:46:10.308803+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:48:43.771 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:07.689+0000 7fe6756e8700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot roll back, removing and adding to missing 2020-03-11T22:48:43.771 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:07.689+0000 7fe6756e8700 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4554.0:1 2020-03-11T22:46:09.150526+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0,28'102 (28'101) modify 1:c156033e:::existing_100:head by client.4560.0:1 2020-03-11T22:46:10.156524+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:48:43.772 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:07.689+0000 7fe6756e8700 20 _merge_object_divergent_entries: keeping 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4554.0:1 2020-03-11T22:46:09.150526+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:48:43.772 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:07.689+0000 7fe6756e8700 20 _merge_object_divergent_entries: keeping 28'102 (28'101) modify 1:c156033e:::existing_100:head by client.4560.0:1 2020-03-11T22:46:10.156524+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:48:43.772 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:07.689+0000 7fe6756e8700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:48:43.772 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:07.689+0000 7fe6756e8700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 28'100 first_divergent_update: 28'101 last_divergent_update: 28'102 2020-03-11T22:48:43.772 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:07.689+0000 7fe6756e8700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2020-03-11T22:48:43.772 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:07.689+0000 7fe6756e8700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:48:43.772 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:07.689+0000 7fe6756e8700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 29'104 2020-03-11T22:48:43.773 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:07.689+0000 7fe6756e8700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 28'102 (28'101) modify 1:c156033e:::existing_100:head by client.4560.0:1 2020-03-11T22:46:10.156524+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:48:43.773 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:07.689+0000 7fe6756e8700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2020-03-11T22:48:43.773 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2020-03-11T22:48:43.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: grep -q '_merge_object_divergent_entries.*rolled back' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:48:43.775 INFO:tasks.workunit.client.0.smithi145.stdout:success 2020-03-11T22:48:43.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:410: TEST_divergent_ec: echo success 2020-03-11T22:48:43.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:412: TEST_divergent_ec: delete_pool test 2020-03-11T22:48:43.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test 2020-03-11T22:48:43.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-03-11T22:48:44.242 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' does not exist 2020-03-11T22:48:44.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:413: TEST_divergent_ec: kill_daemons td/divergent-priors 2020-03-11T22:48:44.250 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:48:44.251 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:48:44.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:48:44.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:48:44.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:48:44.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:48:44.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2020-03-11T22:48:44.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:48:44.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:48:44.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:48:44.359 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:48:44.359 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:48:44.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:48:44.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:48:44.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:48:44.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:48:44.363 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:48:44.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:48:44.364 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:48:44.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:48:44.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:48:44.366 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:48:44.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:48:44.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:48:44.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:48:44.368 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:48:44.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:48:44.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:48:44.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:48:44.382 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:48:44.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:44.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:48:44.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17402 2020-03-11T22:48:44.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:48:44.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:48:44.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: code=0 2020-03-11T22:48:44.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: teardown td/divergent-priors 0 2020-03-11T22:48:44.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:48:44.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-03-11T22:48:44.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:48:44.385 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:48:44.385 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:48:44.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:48:44.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:48:44.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:48:44.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:48:44.387 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:48:44.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:48:44.387 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:48:44.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:48:44.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:48:44.389 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:48:44.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:48:44.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:48:44.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:48:44.391 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:48:44.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:48:44.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-03-11T22:48:44.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:48:44.393 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:48:44.393 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:44.393 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17402 2020-03-11T22:48:44.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17402 2020-03-11T22:48:44.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:48:44.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:48:44.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: return 0 2020-03-11T22:48:44.395 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-03-11T22:48:44.396 INFO:teuthology.orchestra.run.smithi145:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-03-11T22:48:44.463 INFO:tasks.workunit:Running workunit osd/ec-error-rollforward.sh... 2020-03-11T22:48:44.463 INFO:teuthology.orchestra.run.smithi145:workunit test osd/ec-error-rollforward.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4acc1121e47039a3cdf2978ae458c10d153aa141 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh 2020-03-11T22:48:44.491 INFO:tasks.workunit.client.0.smithi145.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-03-11T22:48:44.499 INFO:tasks.workunit.client.0.smithi145.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-03-11T22:48:44.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:48:44.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:48:44.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: export PYTHONWARNINGS=ignore 2020-03-11T22:48:44.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: PYTHONWARNINGS=ignore 2020-03-11T22:48:44.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: export CEPH_CONF=/dev/null 2020-03-11T22:48:44.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: CEPH_CONF=/dev/null 2020-03-11T22:48:44.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: main: unset CEPH_ARGS 2020-03-11T22:48:44.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: local code 2020-03-11T22:48:44.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: run td/ec-error-rollforward 2020-03-11T22:48:44.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:6: run: local dir=td/ec-error-rollforward 2020-03-11T22:48:44.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:7: run: shift 2020-03-11T22:48:44.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: export CEPH_MON=127.0.0.1:7132 2020-03-11T22:48:44.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: CEPH_MON=127.0.0.1:7132 2020-03-11T22:48:44.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:11: run: export CEPH_ARGS 2020-03-11T22:48:44.501 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: uuidgen 2020-03-11T22:48:44.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: CEPH_ARGS+='--fsid=67f07d7f-713e-45eb-a63d-1281efa7fedb --auth-supported=none ' 2020-03-11T22:48:44.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:13: run: CEPH_ARGS+='--mon-host=127.0.0.1:7132 ' 2020-03-11T22:48:44.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: export margin=10 2020-03-11T22:48:44.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: margin=10 2020-03-11T22:48:44.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: export objects=200 2020-03-11T22:48:44.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: objects=200 2020-03-11T22:48:44.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: export poolname=test 2020-03-11T22:48:44.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: poolname=test 2020-03-11T22:48:44.503 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: set 2020-03-11T22:48:44.503 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-03-11T22:48:44.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: local funcs=TEST_ec_error_rollforward 2020-03-11T22:48:44.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:19: run: for func in $funcs 2020-03-11T22:48:44.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:20: run: setup td/ec-error-rollforward 2020-03-11T22:48:44.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ec-error-rollforward 2020-03-11T22:48:44.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ec-error-rollforward 2020-03-11T22:48:44.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2020-03-11T22:48:44.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:48:44.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2020-03-11T22:48:44.507 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:48:44.507 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:48:44.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:48:44.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:48:44.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:48:44.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:48:44.508 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:48:44.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:48:44.508 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:48:44.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:48:44.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:48:44.509 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:48:44.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:48:44.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:48:44.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:48:44.509 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:48:44.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:48:44.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:48:44.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2020-03-11T22:48:44.511 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:48:44.511 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:44.511 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52942 2020-03-11T22:48:44.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.52942 2020-03-11T22:48:44.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:48:44.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:48:44.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ec-error-rollforward 2020-03-11T22:48:44.512 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:48:44.512 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:44.512 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52942 2020-03-11T22:48:44.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.52942 2020-03-11T22:48:44.514 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:48:44.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-03-11T22:48:44.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-03-11T22:48:44.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:48:44.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ec-error-rollforward 1' TERM HUP INT 2020-03-11T22:48:44.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:21: run: TEST_ec_error_rollforward td/ec-error-rollforward 2020-03-11T22:48:44.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:27: TEST_ec_error_rollforward: local dir=td/ec-error-rollforward 2020-03-11T22:48:44.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:28: TEST_ec_error_rollforward: run_mon td/ec-error-rollforward a 2020-03-11T22:48:44.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ec-error-rollforward 2020-03-11T22:48:44.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:48:44.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:48:44.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:48:44.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ec-error-rollforward/a 2020-03-11T22:48:44.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ec-error-rollforward/a --run-dir=td/ec-error-rollforward 2020-03-11T22:48:44.816 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:48:44.816 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:48:44.816 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:48:44.816 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:48:44.816 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:44.817 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52942 2020-03-11T22:48:44.817 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52942/$cluster-$name.asok' 2020-03-11T22:48:44.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ec-error-rollforward/a '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.52942/$cluster-$name.asok' --mon-cluster-log-file=td/ec-error-rollforward/log --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:48:44.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:48:44.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:48:44.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:48:44.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:48:44.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:48:44.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:48:44.969 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:48:44.969 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:48:44.969 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:48:44.969 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:48:44.970 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:44.970 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52942 2020-03-11T22:48:44.970 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52942/ceph-mon.a.asok 2020-03-11T22:48:44.970 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:48:44.970 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.52942/ceph-mon.a.asok config get fsid 2020-03-11T22:48:45.055 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:48:45.055 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:48:45.055 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:48:45.055 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:48:45.056 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:48:45.056 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:48:45.056 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:48:45.056 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:48:45.056 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:48:45.056 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:45.056 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52942 2020-03-11T22:48:45.057 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52942/ceph-mon.a.asok 2020-03-11T22:48:45.057 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:48:45.057 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.52942/ceph-mon.a.asok config get mon_host 2020-03-11T22:48:45.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:29: TEST_ec_error_rollforward: run_mgr td/ec-error-rollforward x 2020-03-11T22:48:45.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/ec-error-rollforward 2020-03-11T22:48:45.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:48:45.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:48:45.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:48:45.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/ec-error-rollforward/x 2020-03-11T22:48:45.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:48:46.016 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:48:46.016 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:48:46.016 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:48:46.016 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:48:46.016 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:46.017 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52942 2020-03-11T22:48:46.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52942/$cluster-$name.asok' 2020-03-11T22:48:46.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:48:46.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ec-error-rollforward/x '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.52942/$cluster-$name.asok' --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:48:46.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:30: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 0 2020-03-11T22:48:46.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/ec-error-rollforward 2020-03-11T22:48:46.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:48:46.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:48:46.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:48:46.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/ec-error-rollforward/0 2020-03-11T22:48:46.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=67f07d7f-713e-45eb-a63d-1281efa7fedb --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:48:46.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:48:46.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:48:46.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:48:46.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2020-03-11T22:48:46.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2020-03-11T22:48:46.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:48:46.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:48:46.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:48:46.042 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:48:46.042 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:48:46.042 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:48:46.043 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:48:46.043 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:46.043 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52942 2020-03-11T22:48:46.043 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52942/$cluster-$name.asok' 2020-03-11T22:48:46.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52942/$cluster-$name.asok' 2020-03-11T22:48:46.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:48:46.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:48:46.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:48:46.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:48:46.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:48:46.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:48:46.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:48:46.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:46.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:48:46.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:48:46.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/ec-error-rollforward/0 2020-03-11T22:48:46.045 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:48:46.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=d600b4b9-f6b4-429e-b709-b35572b85d67 2020-03-11T22:48:46.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 d600b4b9-f6b4-429e-b709-b35572b85d67' 2020-03-11T22:48:46.045 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:48:46.045 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 d600b4b9-f6b4-429e-b709-b35572b85d67 2020-03-11T22:48:46.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDOamleGMv2AhAA33ctZ+NXIOcW/e/czEeA0A== 2020-03-11T22:48:46.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDOamleGMv2AhAA33ctZ+NXIOcW/e/czEeA0A=="}' 2020-03-11T22:48:46.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new d600b4b9-f6b4-429e-b709-b35572b85d67 -i td/ec-error-rollforward/0/new.json 2020-03-11T22:48:46.379 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:48:46.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/ec-error-rollforward/0/new.json 2020-03-11T22:48:46.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=67f07d7f-713e-45eb-a63d-1281efa7fedb --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52942/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDOamleGMv2AhAA33ctZ+NXIOcW/e/czEeA0A== --osd-uuid d600b4b9-f6b4-429e-b709-b35572b85d67 2020-03-11T22:48:46.409 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:48:46.406+0000 7f0b99296ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:46.411 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:48:46.407+0000 7f0b99296ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:46.413 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:48:46.409+0000 7f0b99296ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:46.414 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:48:46.410+0000 7f0b99296ec0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2020-03-11T22:48:46.414 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:48:46.410+0000 7f0b99296ec0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2020-03-11T22:48:46.414 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:48:46.410+0000 7f0b99296ec0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2020-03-11T22:48:46.465 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:48:46.462+0000 7f0b99296ec0 -1 bluestore(td/ec-error-rollforward/0) _read_fsid unparsable uuid 2020-03-11T22:48:49.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/ec-error-rollforward/0/keyring 2020-03-11T22:48:49.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:48:49.411 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2020-03-11T22:48:49.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:48:49.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/ec-error-rollforward/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:48:49.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:48:49.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=67f07d7f-713e-45eb-a63d-1281efa7fedb --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52942/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:49.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:48:49.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:48:49.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:48:49.861 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2020-03-11T22:48:49.877 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:48:49.873+0000 7f8d299dcec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:49.879 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:48:49.875+0000 7f8d299dcec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:49.881 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:48:49.877+0000 7f8d299dcec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:50.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:48:50.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:48:50.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:48:50.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:48:50.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:48:50.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:50.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:48:50.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:50.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:48:50.201 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:48:50.396 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:48:50.393+0000 7f8d299dcec0 -1 Falling back to public interface 2020-03-11T22:48:50.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:48:51.164 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:48:51.161+0000 7f8d299dcec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:48:51.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:48:51.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:51.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:48:51.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:51.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:48:51.546 INFO:tasks.workunit.client.0.smithi145.stdout:1 2020-03-11T22:48:51.942 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/53379,v1:127.0.0.1:6803/53379] [v2:127.0.0.1:6804/53379,v1:127.0.0.1:6805/53379] exists,up d600b4b9-f6b4-429e-b709-b35572b85d67 2020-03-11T22:48:51.942 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 aa2c7391-cb9f-40e7-84c2-86877bcd8365 2020-03-11T22:48:51.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:48:51.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:48:51.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:48:51.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:31: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 1 2020-03-11T22:48:51.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/ec-error-rollforward 2020-03-11T22:48:51.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:48:51.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:48:51.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:48:51.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/ec-error-rollforward/1 2020-03-11T22:48:51.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=67f07d7f-713e-45eb-a63d-1281efa7fedb --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:48:51.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:48:51.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:48:51.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:48:51.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2020-03-11T22:48:51.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2020-03-11T22:48:51.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:48:51.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:48:51.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:48:51.945 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:48:51.945 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:48:51.945 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:48:51.946 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:48:51.946 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:51.946 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52942 2020-03-11T22:48:51.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52942/$cluster-$name.asok' 2020-03-11T22:48:51.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52942/$cluster-$name.asok' 2020-03-11T22:48:51.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:48:51.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:48:51.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:48:51.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:48:51.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:48:51.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:48:51.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:48:51.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:51.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:48:51.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:48:51.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/ec-error-rollforward/1 2020-03-11T22:48:51.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:48:51.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=aa2c7391-cb9f-40e7-84c2-86877bcd8365 2020-03-11T22:48:51.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 aa2c7391-cb9f-40e7-84c2-86877bcd8365' 2020-03-11T22:48:51.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:48:51.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDTamleo0+mOBAAaVVKI0kCtBOl6PIhP3M+cg== 2020-03-11T22:48:51.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDTamleo0+mOBAAaVVKI0kCtBOl6PIhP3M+cg=="}' 2020-03-11T22:48:51.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new aa2c7391-cb9f-40e7-84c2-86877bcd8365 -i td/ec-error-rollforward/1/new.json 2020-03-11T22:48:52.426 INFO:tasks.workunit.client.0.smithi145.stdout:1 2020-03-11T22:48:52.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/ec-error-rollforward/1/new.json 2020-03-11T22:48:52.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=67f07d7f-713e-45eb-a63d-1281efa7fedb --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52942/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDTamleo0+mOBAAaVVKI0kCtBOl6PIhP3M+cg== --osd-uuid aa2c7391-cb9f-40e7-84c2-86877bcd8365 2020-03-11T22:48:52.456 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:48:52.452+0000 7fdb46813ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:52.457 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:48:52.454+0000 7fdb46813ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:52.458 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:48:52.455+0000 7fdb46813ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:52.460 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:48:52.456+0000 7fdb46813ec0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2020-03-11T22:48:52.460 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:48:52.456+0000 7fdb46813ec0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2020-03-11T22:48:52.460 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:48:52.456+0000 7fdb46813ec0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2020-03-11T22:48:52.538 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:48:52.535+0000 7fdb46813ec0 -1 bluestore(td/ec-error-rollforward/1) _read_fsid unparsable uuid 2020-03-11T22:48:55.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/ec-error-rollforward/1/keyring 2020-03-11T22:48:55.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:48:55.525 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2020-03-11T22:48:55.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:48:55.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/ec-error-rollforward/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:48:55.978 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2020-03-11T22:48:55.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:48:55.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:48:55.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:48:55.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:48:55.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=67f07d7f-713e-45eb-a63d-1281efa7fedb --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52942/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:55.996 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:48:55.993+0000 7f0898bd9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:55.998 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:48:55.995+0000 7f0898bd9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:55.999 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:48:55.996+0000 7f0898bd9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:56.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:48:56.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:48:56.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:48:56.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:48:56.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:48:56.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:56.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:48:56.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:48:56.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:56.321 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:48:56.517 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:48:56.514+0000 7f0898bd9ec0 -1 Falling back to public interface 2020-03-11T22:48:56.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:48:57.194 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:48:57.191+0000 7f0898bd9ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:48:57.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:48:57.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:57.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:48:57.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:57.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:48:57.661 INFO:tasks.workunit.client.0.smithi145.stdout:1 2020-03-11T22:48:58.012 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/53734,v1:127.0.0.1:6811/53734] [v2:127.0.0.1:6812/53734,v1:127.0.0.1:6813/53734] exists,up aa2c7391-cb9f-40e7-84c2-86877bcd8365 2020-03-11T22:48:58.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:48:58.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:48:58.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:48:58.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:32: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 2 2020-03-11T22:48:58.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/ec-error-rollforward 2020-03-11T22:48:58.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:48:58.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:48:58.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:48:58.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/ec-error-rollforward/2 2020-03-11T22:48:58.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=67f07d7f-713e-45eb-a63d-1281efa7fedb --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:48:58.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:48:58.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:48:58.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:48:58.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2020-03-11T22:48:58.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2020-03-11T22:48:58.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:48:58.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:48:58.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:48:58.016 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:48:58.016 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:48:58.016 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:48:58.016 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:48:58.016 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:58.017 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52942 2020-03-11T22:48:58.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52942/$cluster-$name.asok' 2020-03-11T22:48:58.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52942/$cluster-$name.asok' 2020-03-11T22:48:58.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:48:58.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:48:58.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:48:58.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:48:58.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:48:58.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:48:58.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:48:58.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:58.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:48:58.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:48:58.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/ec-error-rollforward/2 2020-03-11T22:48:58.019 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:48:58.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=0221e0e3-b936-4b59-92dd-8d8b6ee6f7b7 2020-03-11T22:48:58.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 0221e0e3-b936-4b59-92dd-8d8b6ee6f7b7' 2020-03-11T22:48:58.020 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:48:58.020 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 0221e0e3-b936-4b59-92dd-8d8b6ee6f7b7 2020-03-11T22:48:58.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDaamlelNttARAAK/n6Jz9pNEbCGS+IhxdRFw== 2020-03-11T22:48:58.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDaamlelNttARAAK/n6Jz9pNEbCGS+IhxdRFw=="}' 2020-03-11T22:48:58.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 0221e0e3-b936-4b59-92dd-8d8b6ee6f7b7 -i td/ec-error-rollforward/2/new.json 2020-03-11T22:48:58.440 INFO:tasks.workunit.client.0.smithi145.stdout:2 2020-03-11T22:48:58.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/ec-error-rollforward/2/new.json 2020-03-11T22:48:58.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=67f07d7f-713e-45eb-a63d-1281efa7fedb --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52942/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDaamlelNttARAAK/n6Jz9pNEbCGS+IhxdRFw== --osd-uuid 0221e0e3-b936-4b59-92dd-8d8b6ee6f7b7 2020-03-11T22:48:58.470 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:48:58.467+0000 7fd986f2fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:58.471 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:48:58.469+0000 7fd986f2fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:58.473 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:48:58.470+0000 7fd986f2fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:58.475 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:48:58.472+0000 7fd986f2fec0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2020-03-11T22:48:58.475 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:48:58.472+0000 7fd986f2fec0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2020-03-11T22:48:58.475 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:48:58.472+0000 7fd986f2fec0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2020-03-11T22:48:58.509 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:48:58.506+0000 7fd986f2fec0 -1 bluestore(td/ec-error-rollforward/2) _read_fsid unparsable uuid 2020-03-11T22:49:01.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/ec-error-rollforward/2/keyring 2020-03-11T22:49:01.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:49:01.594 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2020-03-11T22:49:01.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:49:01.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/ec-error-rollforward/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:49:02.045 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2020-03-11T22:49:02.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:49:02.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=67f07d7f-713e-45eb-a63d-1281efa7fedb --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52942/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:02.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:49:02.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:49:02.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:49:02.063 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:02.061+0000 7f916709cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:02.065 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:02.062+0000 7f916709cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:02.066 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:02.063+0000 7f916709cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:02.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:49:02.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:49:02.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:49:02.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:49:02.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:49:02.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:02.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:49:02.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:49:02.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:02.388 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:49:02.582 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:02.580+0000 7f916709cec0 -1 Falling back to public interface 2020-03-11T22:49:02.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:49:03.339 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:03.336+0000 7f916709cec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:49:03.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:49:03.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:03.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:49:03.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:03.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:49:03.727 INFO:tasks.workunit.client.0.smithi145.stdout:1 2020-03-11T22:49:04.070 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/54089,v1:127.0.0.1:6819/54089] [v2:127.0.0.1:6820/54089,v1:127.0.0.1:6821/54089] exists,up 0221e0e3-b936-4b59-92dd-8d8b6ee6f7b7 2020-03-11T22:49:04.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:49:04.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:49:04.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:49:04.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:33: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 3 2020-03-11T22:49:04.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/ec-error-rollforward 2020-03-11T22:49:04.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:49:04.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=3 2020-03-11T22:49:04.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:49:04.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/ec-error-rollforward/3 2020-03-11T22:49:04.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=67f07d7f-713e-45eb-a63d-1281efa7fedb --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:49:04.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:49:04.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:49:04.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:49:04.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2020-03-11T22:49:04.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2020-03-11T22:49:04.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:49:04.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:49:04.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:49:04.073 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:49:04.073 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:49:04.073 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:49:04.074 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:49:04.074 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:04.074 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52942 2020-03-11T22:49:04.074 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52942/$cluster-$name.asok' 2020-03-11T22:49:04.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52942/$cluster-$name.asok' 2020-03-11T22:49:04.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:49:04.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:49:04.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:49:04.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:49:04.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:49:04.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:49:04.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:49:04.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:04.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:49:04.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:49:04.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/ec-error-rollforward/3 2020-03-11T22:49:04.076 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:49:04.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=8930aee0-dbce-44f0-872c-de9f2335f752 2020-03-11T22:49:04.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd3 8930aee0-dbce-44f0-872c-de9f2335f752' 2020-03-11T22:49:04.077 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:49:04.077 INFO:tasks.workunit.client.0.smithi145.stdout:add osd3 8930aee0-dbce-44f0-872c-de9f2335f752 2020-03-11T22:49:04.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDgamleAtHpBBAAnHaJFDbyeJxE2T95Z2LH4g== 2020-03-11T22:49:04.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDgamleAtHpBBAAnHaJFDbyeJxE2T95Z2LH4g=="}' 2020-03-11T22:49:04.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 8930aee0-dbce-44f0-872c-de9f2335f752 -i td/ec-error-rollforward/3/new.json 2020-03-11T22:49:04.609 INFO:tasks.workunit.client.0.smithi145.stdout:3 2020-03-11T22:49:04.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/ec-error-rollforward/3/new.json 2020-03-11T22:49:04.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 3 --fsid=67f07d7f-713e-45eb-a63d-1281efa7fedb --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52942/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDgamleAtHpBBAAnHaJFDbyeJxE2T95Z2LH4g== --osd-uuid 8930aee0-dbce-44f0-872c-de9f2335f752 2020-03-11T22:49:04.640 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:04.637+0000 7ff48079eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:04.667 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:04.665+0000 7ff48079eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:04.669 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:04.666+0000 7ff48079eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:04.669 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:04.666+0000 7ff48079eec0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2020-03-11T22:49:04.669 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:04.666+0000 7ff48079eec0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2020-03-11T22:49:04.670 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:04.666+0000 7ff48079eec0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2020-03-11T22:49:04.726 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:04.723+0000 7ff48079eec0 -1 bluestore(td/ec-error-rollforward/3) _read_fsid unparsable uuid 2020-03-11T22:49:07.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/ec-error-rollforward/3/keyring 2020-03-11T22:49:07.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:49:07.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd3 key to auth repository 2020-03-11T22:49:07.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/ec-error-rollforward/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:49:07.837 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd3 key to auth repository 2020-03-11T22:49:08.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.3 2020-03-11T22:49:08.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:49:08.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 3 --fsid=67f07d7f-713e-45eb-a63d-1281efa7fedb --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52942/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:08.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:49:08.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:49:08.288 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.3 2020-03-11T22:49:08.305 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:08.303+0000 7fae302e9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:08.307 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:08.304+0000 7fae302e9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:08.308 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:08.306+0000 7fae302e9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:08.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 3 2020-03-11T22:49:08.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:49:08.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=3 2020-03-11T22:49:08.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:49:08.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:49:08.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:08.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:49:08.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:08.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:49:08.629 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:49:08.824 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:08.821+0000 7fae302e9ec0 -1 Falling back to public interface 2020-03-11T22:49:08.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:49:09.600 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:09.597+0000 7fae302e9ec0 -1 osd.3 0 log_to_monitors {default=true} 2020-03-11T22:49:09.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:49:09.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:09.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:49:09.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:09.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:49:09.971 INFO:tasks.workunit.client.0.smithi145.stdout:1 2020-03-11T22:49:10.318 INFO:tasks.workunit.client.0.smithi145.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/54444,v1:127.0.0.1:6827/54444] [v2:127.0.0.1:6828/54444,v1:127.0.0.1:6829/54444] exists,up 8930aee0-dbce-44f0-872c-de9f2335f752 2020-03-11T22:49:10.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:49:10.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:49:10.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:49:10.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:35: TEST_ec_error_rollforward: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2020-03-11T22:49:10.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:36: TEST_ec_error_rollforward: ceph osd pool create ec 1 1 erasure ec-profile 2020-03-11T22:49:11.441 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'ec' already exists 2020-03-11T22:49:11.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:38: TEST_ec_error_rollforward: rados -p ec put foo /etc/passwd 2020-03-11T22:49:12.848 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: cat td/ec-error-rollforward/osd.2.pid 2020-03-11T22:49:12.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: kill -STOP 54089 2020-03-11T22:49:12.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:43: TEST_ec_error_rollforward: pids=54699 2020-03-11T22:49:12.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:44: TEST_ec_error_rollforward: sleep 1 2020-03-11T22:49:12.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:42: TEST_ec_error_rollforward: rados -p ec rm foo 2020-03-11T22:49:13.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:46: TEST_ec_error_rollforward: pids+=' 54722' 2020-03-11T22:49:13.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:45: TEST_ec_error_rollforward: rados -p ec rm a 2020-03-11T22:49:13.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:48: TEST_ec_error_rollforward: pids+=' 54723' 2020-03-11T22:49:13.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:47: TEST_ec_error_rollforward: rados -p ec rm b 2020-03-11T22:49:13.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:50: TEST_ec_error_rollforward: pids+=' 54724' 2020-03-11T22:49:13.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:51: TEST_ec_error_rollforward: sleep 1 2020-03-11T22:49:13.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:49: TEST_ec_error_rollforward: rados -p ec rm c 2020-03-11T22:49:14.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:54: TEST_ec_error_rollforward: kill_daemons td/ec-error-rollforward KILL osd 2020-03-11T22:49:14.853 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:49:14.853 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:49:14.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:49:14.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:49:14.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:49:14.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:49:14.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:55: TEST_ec_error_rollforward: kill 54699 54722 54723 54724 2020-03-11T22:49:14.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:56: TEST_ec_error_rollforward: wait 2020-03-11T22:49:14.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:58: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 0 2020-03-11T22:49:14.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/ec-error-rollforward 2020-03-11T22:49:14.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:49:14.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:49:14.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:49:14.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/ec-error-rollforward/0 2020-03-11T22:49:14.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=67f07d7f-713e-45eb-a63d-1281efa7fedb --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:49:14.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:49:14.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:49:14.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:49:14.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2020-03-11T22:49:14.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2020-03-11T22:49:14.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:49:14.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:49:14.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:49:14.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:49:14.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:49:14.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:49:14.969 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:49:14.970 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:14.970 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52942 2020-03-11T22:49:14.970 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52942/$cluster-$name.asok' 2020-03-11T22:49:14.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52942/$cluster-$name.asok' 2020-03-11T22:49:14.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:49:14.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:49:14.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:49:14.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:49:14.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:49:14.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:14.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:49:14.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:49:14.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/ec-error-rollforward/0 2020-03-11T22:49:14.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:49:14.972 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2020-03-11T22:49:14.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=67f07d7f-713e-45eb-a63d-1281efa7fedb --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52942/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:14.973 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/ec-error-rollforward/0/whoami 2020-03-11T22:49:14.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:49:14.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:49:14.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:49:14.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:49:14.988 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:14.985+0000 7fe3e47c6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:14.990 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:14.987+0000 7fe3e47c6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:14.991 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:14.988+0000 7fe3e47c6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:15.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:49:15.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:49:15.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:49:15.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:49:15.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:49:15.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:15.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:49:15.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:15.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:49:15.325 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:49:15.507 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:15.504+0000 7fe3e47c6ec0 -1 Falling back to public interface 2020-03-11T22:49:15.674 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/53379,v1:127.0.0.1:6803/53379] [v2:127.0.0.1:6804/53379,v1:127.0.0.1:6805/53379] exists,up d600b4b9-f6b4-429e-b709-b35572b85d67 2020-03-11T22:49:15.674 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2020-03-11T22:49:15.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:49:15.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:49:15.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:49:15.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:59: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 1 2020-03-11T22:49:15.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/ec-error-rollforward 2020-03-11T22:49:15.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:49:15.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:49:15.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:49:15.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/ec-error-rollforward/1 2020-03-11T22:49:15.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=67f07d7f-713e-45eb-a63d-1281efa7fedb --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:49:15.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:49:15.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:49:15.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:49:15.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2020-03-11T22:49:15.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2020-03-11T22:49:15.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:49:15.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:49:15.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:49:15.677 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:49:15.677 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:49:15.677 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:49:15.678 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:49:15.678 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:15.678 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52942 2020-03-11T22:49:15.678 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52942/$cluster-$name.asok' 2020-03-11T22:49:15.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52942/$cluster-$name.asok' 2020-03-11T22:49:15.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:49:15.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:49:15.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:49:15.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:49:15.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:49:15.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:15.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:49:15.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:49:15.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/ec-error-rollforward/1 2020-03-11T22:49:15.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:49:15.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=67f07d7f-713e-45eb-a63d-1281efa7fedb --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52942/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:15.680 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/ec-error-rollforward/1/whoami 2020-03-11T22:49:15.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:49:15.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:49:15.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:49:15.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:49:15.691 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:15.688+0000 7f63ac905ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:15.692 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:15.690+0000 7f63ac905ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:15.693 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:15.691+0000 7f63ac905ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:16.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-03-11T22:49:16.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:49:16.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:49:16.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:49:16.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:49:16.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:16.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:49:16.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:16.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:49:16.020 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:49:16.207 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:16.205+0000 7f63ac905ec0 -1 Falling back to public interface 2020-03-11T22:49:16.364 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 9 up_thru 22 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/53734,v1:127.0.0.1:6811/53734] [v2:127.0.0.1:6812/53734,v1:127.0.0.1:6813/53734] exists,up aa2c7391-cb9f-40e7-84c2-86877bcd8365 2020-03-11T22:49:16.364 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2020-03-11T22:49:16.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:49:16.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:49:16.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:49:16.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:60: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 2 2020-03-11T22:49:16.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/ec-error-rollforward 2020-03-11T22:49:16.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:49:16.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:49:16.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:49:16.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/ec-error-rollforward/2 2020-03-11T22:49:16.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=67f07d7f-713e-45eb-a63d-1281efa7fedb --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:49:16.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:49:16.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:49:16.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:49:16.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2020-03-11T22:49:16.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2020-03-11T22:49:16.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:49:16.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:49:16.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:49:16.367 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:49:16.367 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:49:16.367 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:49:16.367 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:49:16.368 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:16.368 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52942 2020-03-11T22:49:16.368 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52942/$cluster-$name.asok' 2020-03-11T22:49:16.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52942/$cluster-$name.asok' 2020-03-11T22:49:16.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:49:16.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:49:16.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:49:16.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:49:16.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:49:16.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:16.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:49:16.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:49:16.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/ec-error-rollforward/2 2020-03-11T22:49:16.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:49:16.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=67f07d7f-713e-45eb-a63d-1281efa7fedb --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52942/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:16.370 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/ec-error-rollforward/2/whoami 2020-03-11T22:49:16.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:49:16.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:49:16.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:49:16.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:49:16.382 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:16.379+0000 7eff44819ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:16.384 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:16.381+0000 7eff44819ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:16.385 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:16.382+0000 7eff44819ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:16.474 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:16.472+0000 7fe3e47c6ec0 -1 osd.0 24 log_to_monitors {default=true} 2020-03-11T22:49:16.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:49:16.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:49:16.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:49:16.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:49:16.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:49:16.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:16.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:49:16.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:16.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:49:16.711 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:49:16.905 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:16.902+0000 7eff44819ec0 -1 Falling back to public interface 2020-03-11T22:49:17.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:49:17.219 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:17.216+0000 7f63ac905ec0 -1 osd.1 24 log_to_monitors {default=true} 2020-03-11T22:49:18.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:49:18.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:18.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:49:18.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:18.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:49:18.084 INFO:tasks.workunit.client.0.smithi145.stdout:1 2020-03-11T22:49:18.120 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:18.118+0000 7eff44819ec0 -1 osd.2 23 log_to_monitors {default=true} 2020-03-11T22:49:18.458 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 31 up_thru 0 down_at 27 last_clean_interval [14,23) [v2:127.0.0.1:6818/55048,v1:127.0.0.1:6819/55048] [v2:127.0.0.1:6820/55048,v1:127.0.0.1:6821/55048] exists,up 0221e0e3-b936-4b59-92dd-8d8b6ee6f7b7 2020-03-11T22:49:18.458 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.3 2020-03-11T22:49:18.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:49:18.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:49:18.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:49:18.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:61: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 3 2020-03-11T22:49:18.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/ec-error-rollforward 2020-03-11T22:49:18.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:49:18.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=3 2020-03-11T22:49:18.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:49:18.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/ec-error-rollforward/3 2020-03-11T22:49:18.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=67f07d7f-713e-45eb-a63d-1281efa7fedb --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:49:18.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:49:18.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:49:18.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:49:18.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2020-03-11T22:49:18.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2020-03-11T22:49:18.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:49:18.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:49:18.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:49:18.461 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:49:18.461 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:49:18.461 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:49:18.461 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:49:18.462 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:18.462 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52942 2020-03-11T22:49:18.462 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52942/$cluster-$name.asok' 2020-03-11T22:49:18.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52942/$cluster-$name.asok' 2020-03-11T22:49:18.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:49:18.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:49:18.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:49:18.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:49:18.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:49:18.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:18.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:49:18.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:49:18.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/ec-error-rollforward/3 2020-03-11T22:49:18.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.3 2020-03-11T22:49:18.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 3 --fsid=67f07d7f-713e-45eb-a63d-1281efa7fedb --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52942/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:18.464 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/ec-error-rollforward/3/whoami 2020-03-11T22:49:18.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 3 = 3 ']' 2020-03-11T22:49:18.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:49:18.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:49:18.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:49:18.475 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:18.472+0000 7fb65ca3aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:18.477 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:18.474+0000 7fb65ca3aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:18.479 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:18.476+0000 7fb65ca3aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:18.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 3 2020-03-11T22:49:18.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:49:18.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=3 2020-03-11T22:49:18.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:49:18.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:49:18.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:18.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:49:18.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:18.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:49:18.800 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:49:18.999 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:18.996+0000 7fb65ca3aec0 -1 Falling back to public interface 2020-03-11T22:49:19.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:49:19.947 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:19.944+0000 7fb65ca3aec0 -1 osd.3 24 log_to_monitors {default=true} 2020-03-11T22:49:20.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:49:20.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:20.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:49:20.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:20.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:49:20.230 INFO:tasks.workunit.client.0.smithi145.stdout:1 2020-03-11T22:49:20.587 INFO:tasks.workunit.client.0.smithi145.stdout:osd.3 up in weight 1 up_from 33 up_thru 0 down_at 27 last_clean_interval [19,24) [v2:127.0.0.1:6826/55355,v1:127.0.0.1:6827/55355] [v2:127.0.0.1:6828/55355,v1:127.0.0.1:6829/55355] exists,up 8930aee0-dbce-44f0-872c-de9f2335f752 2020-03-11T22:49:20.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:49:20.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:49:20.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:49:20.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:63: TEST_ec_error_rollforward: wait_for_clean 2020-03-11T22:49:20.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:49:20.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:49:20.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:49:20.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:49:20.588 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:49:20.588 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:49:20.588 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:49:20.588 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:49:20.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:49:20.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:49:20.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:49:20.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:49:20.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:49:20.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:49:20.676 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:49:21.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:49:21.013 INFO:tasks.workunit.client.0.smithi145.stderr:1 2020-03-11T22:49:21.013 INFO:tasks.workunit.client.0.smithi145.stderr:2 2020-03-11T22:49:21.013 INFO:tasks.workunit.client.0.smithi145.stderr:3' 2020-03-11T22:49:21.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:49:21.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:21.014 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:49:21.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=111669149699 2020-03-11T22:49:21.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-111669149699' 2020-03-11T22:49:21.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:21.134 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:49:21.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=124554051587 2020-03-11T22:49:21.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-111669149699 1-124554051587' 2020-03-11T22:49:21.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:21.257 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:49:21.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=133143986179 2020-03-11T22:49:21.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-111669149699 1-124554051587 2-133143986179' 2020-03-11T22:49:21.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:21.378 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:49:21.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=141733920771 2020-03-11T22:49:21.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-111669149699 1-124554051587 2-133143986179 3-141733920771' 2020-03-11T22:49:21.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:21.500 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-111669149699 2020-03-11T22:49:21.500 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:21.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:49:21.502 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-111669149699 2020-03-11T22:49:21.502 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:21.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=111669149699 2020-03-11T22:49:21.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 111669149699' 2020-03-11T22:49:21.503 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 111669149699 2020-03-11T22:49:21.503 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:49:21.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 111669149699 -lt 111669149699 2020-03-11T22:49:21.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:21.840 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-124554051587 2020-03-11T22:49:21.840 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:21.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:49:21.842 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-124554051587 2020-03-11T22:49:21.842 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:21.843 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 124554051587 2020-03-11T22:49:21.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=124554051587 2020-03-11T22:49:21.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 124554051587' 2020-03-11T22:49:21.844 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:49:22.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 124554051587 -lt 124554051587 2020-03-11T22:49:22.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:22.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-133143986179 2020-03-11T22:49:22.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:22.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:49:22.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-133143986179 2020-03-11T22:49:22.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:22.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=133143986179 2020-03-11T22:49:22.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 133143986179' 2020-03-11T22:49:22.187 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 133143986179 2020-03-11T22:49:22.188 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:49:22.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 1 -lt 133143986179 2020-03-11T22:49:22.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:49:23.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:49:23.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:49:23.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 133143986180 -lt 133143986179 2020-03-11T22:49:23.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:23.872 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-141733920771 2020-03-11T22:49:23.872 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:23.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:49:23.874 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-141733920771 2020-03-11T22:49:23.874 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:23.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=141733920771 2020-03-11T22:49:23.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 141733920771' 2020-03-11T22:49:23.876 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 141733920771 2020-03-11T22:49:23.876 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:49:24.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 141733920771 -lt 141733920771 2020-03-11T22:49:24.241 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:49:24.242 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:49:24.242 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:49:24.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:49:24.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:49:24.684 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:49:24.684 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:49:24.684 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:49:24.684 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:49:24.684 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:49:24.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:49:25.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:49:25.013 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:49:25.013 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:49:25.013 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:49:25.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:49:25.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:49:25.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:49:25.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:22: run: teardown td/ec-error-rollforward 2020-03-11T22:49:25.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2020-03-11T22:49:25.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:49:25.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2020-03-11T22:49:25.453 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:49:25.453 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:49:25.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:49:25.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:49:25.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:49:25.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:49:25.562 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:49:25.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:49:25.563 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:49:25.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:49:25.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:49:25.564 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:49:25.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:49:25.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:49:25.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:49:25.566 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:49:25.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:49:25.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:49:25.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2020-03-11T22:49:25.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:49:25.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:25.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52942 2020-03-11T22:49:25.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.52942 2020-03-11T22:49:25.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:49:25.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:49:25.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: code=0 2020-03-11T22:49:25.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: teardown td/ec-error-rollforward 0 2020-03-11T22:49:25.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2020-03-11T22:49:25.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-03-11T22:49:25.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2020-03-11T22:49:25.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:49:25.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:49:25.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:49:25.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:49:25.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:49:25.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:49:25.583 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:49:25.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:49:25.583 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:49:25.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:49:25.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:49:25.584 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:49:25.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:49:25.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:49:25.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:49:25.584 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:49:25.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:49:25.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-03-11T22:49:25.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2020-03-11T22:49:25.586 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:49:25.586 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:25.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52942 2020-03-11T22:49:25.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.52942 2020-03-11T22:49:25.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:49:25.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:49:25.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: return 0 2020-03-11T22:49:25.587 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-03-11T22:49:25.588 INFO:teuthology.orchestra.run.smithi145:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-03-11T22:49:25.662 INFO:tasks.workunit:Running workunit osd/osd-backfill-prio.sh... 2020-03-11T22:49:25.662 INFO:teuthology.orchestra.run.smithi145:workunit test osd/osd-backfill-prio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4acc1121e47039a3cdf2978ae458c10d153aa141 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh 2020-03-11T22:49:25.713 INFO:tasks.workunit.client.0.smithi145.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-03-11T22:49:25.723 INFO:tasks.workunit.client.0.smithi145.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-03-11T22:49:25.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:49:25.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:49:25.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: export PYTHONWARNINGS=ignore 2020-03-11T22:49:25.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: PYTHONWARNINGS=ignore 2020-03-11T22:49:25.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: export CEPH_CONF=/dev/null 2020-03-11T22:49:25.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: CEPH_CONF=/dev/null 2020-03-11T22:49:25.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: main: unset CEPH_ARGS 2020-03-11T22:49:25.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: local code 2020-03-11T22:49:25.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: run td/osd-backfill-prio 2020-03-11T22:49:25.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:21: run: local dir=td/osd-backfill-prio 2020-03-11T22:49:25.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:22: run: shift 2020-03-11T22:49:25.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:25: run: export CEPH_MON=127.0.0.1:7114 2020-03-11T22:49:25.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:25: run: CEPH_MON=127.0.0.1:7114 2020-03-11T22:49:25.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:26: run: export CEPH_ARGS 2020-03-11T22:49:25.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:27: run: uuidgen 2020-03-11T22:49:25.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:27: run: CEPH_ARGS+='--fsid=9ff197a8-fd34-42fc-bffc-e43264b5b9b7 --auth-supported=none ' 2020-03-11T22:49:25.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:49:25.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:29: run: CEPH_ARGS+='--osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:49:25.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:30: run: export objects=50 2020-03-11T22:49:25.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:30: run: objects=50 2020-03-11T22:49:25.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:31: run: export poolprefix=test 2020-03-11T22:49:25.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:31: run: poolprefix=test 2020-03-11T22:49:25.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:32: run: export FORCE_PRIO=254 2020-03-11T22:49:25.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:32: run: FORCE_PRIO=254 2020-03-11T22:49:25.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:33: run: export DEGRADED_PRIO=150 2020-03-11T22:49:25.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:33: run: DEGRADED_PRIO=150 2020-03-11T22:49:25.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:34: run: export NORMAL_PRIO=110 2020-03-11T22:49:25.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:34: run: NORMAL_PRIO=110 2020-03-11T22:49:25.727 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:36: run: set 2020-03-11T22:49:25.727 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:36: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-03-11T22:49:25.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:36: run: local 'funcs=TEST_backfill_pool_priority 2020-03-11T22:49:25.730 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_backfill_priority' 2020-03-11T22:49:25.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:37: run: for func in $funcs 2020-03-11T22:49:25.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:38: run: setup td/osd-backfill-prio 2020-03-11T22:49:25.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-prio 2020-03-11T22:49:25.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-prio 2020-03-11T22:49:25.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-03-11T22:49:25.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:49:25.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-03-11T22:49:25.731 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:49:25.731 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:49:25.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:49:25.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:49:25.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:49:25.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:49:25.732 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:49:25.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:49:25.732 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:49:25.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:49:25.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:49:25.732 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:49:25.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:49:25.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:49:25.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:49:25.733 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:49:25.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:49:25.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:49:25.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-03-11T22:49:25.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:49:25.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:25.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56096 2020-03-11T22:49:25.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.56096 2020-03-11T22:49:25.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:49:25.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:49:25.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-prio 2020-03-11T22:49:25.737 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:49:25.737 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:25.737 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56096 2020-03-11T22:49:25.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.56096 2020-03-11T22:49:25.738 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:49:25.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-03-11T22:49:25.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-03-11T22:49:25.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:49:25.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-prio 1' TERM HUP INT 2020-03-11T22:49:25.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:39: run: TEST_backfill_pool_priority td/osd-backfill-prio 2020-03-11T22:49:25.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:326: TEST_backfill_pool_priority: local dir=td/osd-backfill-prio 2020-03-11T22:49:25.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:327: TEST_backfill_pool_priority: local pools=3 2020-03-11T22:49:25.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:328: TEST_backfill_pool_priority: local OSDS=2 2020-03-11T22:49:25.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:330: TEST_backfill_pool_priority: run_mon td/osd-backfill-prio a 2020-03-11T22:49:25.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-prio 2020-03-11T22:49:25.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:49:25.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:49:25.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:49:25.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-prio/a 2020-03-11T22:49:25.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-prio/a --run-dir=td/osd-backfill-prio 2020-03-11T22:49:26.030 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:49:26.030 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:49:26.030 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:49:26.031 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:49:26.031 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:26.031 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56096 2020-03-11T22:49:26.031 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56096/$cluster-$name.asok' 2020-03-11T22:49:26.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-prio/a '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.56096/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-prio/log --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:49:26.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:49:26.199 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:49:26.199 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:49:26.200 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:49:26.200 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:49:26.200 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:49:26.200 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:49:26.200 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:49:26.200 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:49:26.201 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:49:26.201 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:26.201 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56096 2020-03-11T22:49:26.201 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56096/ceph-mon.a.asok 2020-03-11T22:49:26.201 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:49:26.201 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.56096/ceph-mon.a.asok config get fsid 2020-03-11T22:49:26.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:49:26.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:49:26.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:49:26.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:49:26.286 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:49:26.286 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:49:26.286 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:49:26.286 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:49:26.286 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:49:26.287 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:26.287 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56096 2020-03-11T22:49:26.287 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56096/ceph-mon.a.asok 2020-03-11T22:49:26.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:49:26.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.56096/ceph-mon.a.asok config get mon_host 2020-03-11T22:49:26.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:331: TEST_backfill_pool_priority: run_mgr td/osd-backfill-prio x 2020-03-11T22:49:26.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-backfill-prio 2020-03-11T22:49:26.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:49:26.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:49:26.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:49:26.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-backfill-prio/x 2020-03-11T22:49:26.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:49:27.247 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:49:27.248 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:49:27.248 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:49:27.248 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:49:27.249 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:27.249 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56096 2020-03-11T22:49:27.249 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56096/$cluster-$name.asok' 2020-03-11T22:49:27.249 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:49:27.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-prio/x '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.56096/$cluster-$name.asok' --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:49:27.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:332: TEST_backfill_pool_priority: export CEPH_ARGS 2020-03-11T22:49:27.273 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: expr 2 - 1 2020-03-11T22:49:27.274 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: seq 0 1 2020-03-11T22:49:27.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:49:27.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:336: TEST_backfill_pool_priority: run_osd td/osd-backfill-prio 0 2020-03-11T22:49:27.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:49:27.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:49:27.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:49:27.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:49:27.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/0 2020-03-11T22:49:27.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=9ff197a8-fd34-42fc-bffc-e43264b5b9b7 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:49:27.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:49:27.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:49:27.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:49:27.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/0' 2020-03-11T22:49:27.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/0/journal' 2020-03-11T22:49:27.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:49:27.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:49:27.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:49:27.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:49:27.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:49:27.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:49:27.281 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:49:27.281 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:27.281 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56096 2020-03-11T22:49:27.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56096/$cluster-$name.asok' 2020-03-11T22:49:27.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56096/$cluster-$name.asok' 2020-03-11T22:49:27.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:49:27.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:49:27.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:49:27.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:49:27.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:49:27.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:49:27.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:49:27.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:27.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:49:27.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:49:27.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/0 2020-03-11T22:49:27.284 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:49:27.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=9fa9d7f9-51e4-441a-b173-2a43a682b2ab 2020-03-11T22:49:27.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 9fa9d7f9-51e4-441a-b173-2a43a682b2ab' 2020-03-11T22:49:27.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:49:27.285 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 9fa9d7f9-51e4-441a-b173-2a43a682b2ab 2020-03-11T22:49:27.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQD3amleKgJlERAAP/cUovg3V9XRhS0pfewtkg== 2020-03-11T22:49:27.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQD3amleKgJlERAAP/cUovg3V9XRhS0pfewtkg=="}' 2020-03-11T22:49:27.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 9fa9d7f9-51e4-441a-b173-2a43a682b2ab -i td/osd-backfill-prio/0/new.json 2020-03-11T22:49:27.637 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:49:27.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/0/new.json 2020-03-11T22:49:27.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=9ff197a8-fd34-42fc-bffc-e43264b5b9b7 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56096/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD3amleKgJlERAAP/cUovg3V9XRhS0pfewtkg== --osd-uuid 9fa9d7f9-51e4-441a-b173-2a43a682b2ab 2020-03-11T22:49:27.673 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:27.669+0000 7fa34a053ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:27.674 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:27.671+0000 7fa34a053ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:27.677 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:27.673+0000 7fa34a053ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:27.677 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:27.673+0000 7fa34a053ec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:49:27.677 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:27.673+0000 7fa34a053ec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:49:27.678 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:27.673+0000 7fa34a053ec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:49:27.721 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:27.717+0000 7fa34a053ec0 -1 bluestore(td/osd-backfill-prio/0) _read_fsid unparsable uuid 2020-03-11T22:49:30.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/0/keyring 2020-03-11T22:49:30.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:49:30.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:49:30.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:49:30.689 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2020-03-11T22:49:31.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:49:31.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=9ff197a8-fd34-42fc-bffc-e43264b5b9b7 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56096/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:31.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:49:31.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:49:31.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:49:31.139 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2020-03-11T22:49:31.162 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:31.158+0000 7fa06bf0eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:31.164 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:31.160+0000 7fa06bf0eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:31.165 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:31.162+0000 7fa06bf0eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:31.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:49:31.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:49:31.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:49:31.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:49:31.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:49:31.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:31.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:49:31.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:31.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:49:31.487 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:49:31.681 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:31.677+0000 7fa06bf0eec0 -1 Falling back to public interface 2020-03-11T22:49:31.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:49:32.420 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:32.416+0000 7fa06bf0eec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:49:32.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:49:32.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:32.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:49:32.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:32.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:49:32.841 INFO:tasks.workunit.client.0.smithi145.stdout:1 2020-03-11T22:49:33.201 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/56528,v1:127.0.0.1:6803/56528] [v2:127.0.0.1:6804/56528,v1:127.0.0.1:6805/56528] exists,up 9fa9d7f9-51e4-441a-b173-2a43a682b2ab 2020-03-11T22:49:33.201 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 17a1f19c-61d6-4dff-abf1-fe8c84001054 2020-03-11T22:49:33.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:49:33.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:49:33.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:49:33.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:49:33.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:336: TEST_backfill_pool_priority: run_osd td/osd-backfill-prio 1 2020-03-11T22:49:33.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:49:33.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:49:33.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:49:33.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:49:33.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/1 2020-03-11T22:49:33.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=9ff197a8-fd34-42fc-bffc-e43264b5b9b7 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:49:33.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:49:33.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:49:33.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:49:33.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/1' 2020-03-11T22:49:33.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/1/journal' 2020-03-11T22:49:33.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:49:33.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:49:33.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:49:33.204 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:49:33.204 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:49:33.204 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:49:33.204 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:49:33.204 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:33.204 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56096 2020-03-11T22:49:33.204 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56096/$cluster-$name.asok' 2020-03-11T22:49:33.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56096/$cluster-$name.asok' 2020-03-11T22:49:33.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:49:33.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:49:33.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:49:33.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:49:33.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:49:33.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:49:33.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:49:33.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:33.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:49:33.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:49:33.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/1 2020-03-11T22:49:33.206 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:49:33.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=17a1f19c-61d6-4dff-abf1-fe8c84001054 2020-03-11T22:49:33.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 17a1f19c-61d6-4dff-abf1-fe8c84001054' 2020-03-11T22:49:33.207 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:49:33.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQD9amlec5awDBAAPa63I7qRz15LIvYszqtwLg== 2020-03-11T22:49:33.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQD9amlec5awDBAAPa63I7qRz15LIvYszqtwLg=="}' 2020-03-11T22:49:33.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 17a1f19c-61d6-4dff-abf1-fe8c84001054 -i td/osd-backfill-prio/1/new.json 2020-03-11T22:49:33.693 INFO:tasks.workunit.client.0.smithi145.stdout:1 2020-03-11T22:49:33.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/1/new.json 2020-03-11T22:49:33.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=9ff197a8-fd34-42fc-bffc-e43264b5b9b7 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56096/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD9amlec5awDBAAPa63I7qRz15LIvYszqtwLg== --osd-uuid 17a1f19c-61d6-4dff-abf1-fe8c84001054 2020-03-11T22:49:33.729 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:33.725+0000 7f666daa2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:33.731 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:33.727+0000 7f666daa2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:33.733 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:33.729+0000 7f666daa2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:33.733 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:33.730+0000 7f666daa2ec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:49:33.734 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:33.730+0000 7f666daa2ec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:49:33.734 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:33.730+0000 7f666daa2ec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:49:33.785 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:33.781+0000 7f666daa2ec0 -1 bluestore(td/osd-backfill-prio/1) _read_fsid unparsable uuid 2020-03-11T22:49:36.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/1/keyring 2020-03-11T22:49:36.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:49:36.828 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2020-03-11T22:49:36.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:49:36.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:49:37.288 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2020-03-11T22:49:37.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:49:37.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=9ff197a8-fd34-42fc-bffc-e43264b5b9b7 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56096/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:37.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:49:37.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:49:37.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:49:37.311 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:37.307+0000 7fe106aadec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:37.313 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:37.309+0000 7fe106aadec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:37.314 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:37.310+0000 7fe106aadec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:37.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:49:37.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:49:37.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:49:37.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:49:37.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:49:37.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:37.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:49:37.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:37.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:49:37.634 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:49:37.829 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:37.825+0000 7fe106aadec0 -1 Falling back to public interface 2020-03-11T22:49:37.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:49:38.586 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:49:38.583+0000 7fe106aadec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:49:38.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:49:38.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:38.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:49:38.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:38.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:49:38.982 INFO:tasks.workunit.client.0.smithi145.stdout:1 2020-03-11T22:49:39.365 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/56883,v1:127.0.0.1:6811/56883] [v2:127.0.0.1:6812/56883,v1:127.0.0.1:6813/56883] exists,up 17a1f19c-61d6-4dff-abf1-fe8c84001054 2020-03-11T22:49:39.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:49:39.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:49:39.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:49:39.366 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: seq 1 3 2020-03-11T22:49:39.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:39.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:341: TEST_backfill_pool_priority: create_pool test1 1 1 2020-03-11T22:49:39.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test1 1 1 2020-03-11T22:49:39.976 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test1' already exists 2020-03-11T22:49:39.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:49:40.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: ceph osd pool set test1 size 2 2020-03-11T22:49:41.703 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 2 2020-03-11T22:49:41.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:41.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:341: TEST_backfill_pool_priority: create_pool test2 1 1 2020-03-11T22:49:41.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test2 1 1 2020-03-11T22:49:42.174 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test2' already exists 2020-03-11T22:49:42.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:49:43.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: ceph osd pool set test2 size 2 2020-03-11T22:49:43.874 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 2 size to 2 2020-03-11T22:49:43.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:43.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:341: TEST_backfill_pool_priority: create_pool test3 1 1 2020-03-11T22:49:43.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test3 1 1 2020-03-11T22:49:44.367 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test3' already exists 2020-03-11T22:49:44.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:49:45.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: ceph osd pool set test3 size 2 2020-03-11T22:49:46.145 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 3 size to 2 2020-03-11T22:49:46.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:344: TEST_backfill_pool_priority: sleep 5 2020-03-11T22:49:51.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:346: TEST_backfill_pool_priority: wait_for_clean 2020-03-11T22:49:51.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:49:51.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:49:51.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:49:51.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:49:51.156 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:49:51.156 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:49:51.156 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:49:51.156 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:49:51.156 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:49:51.156 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:49:51.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:49:51.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:49:51.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:49:51.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:49:51.259 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:49:51.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:49:51.621 INFO:tasks.workunit.client.0.smithi145.stderr:1' 2020-03-11T22:49:51.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:49:51.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:51.621 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:49:51.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836487 2020-03-11T22:49:51.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836487' 2020-03-11T22:49:51.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:51.748 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:49:51.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705669 2020-03-11T22:49:51.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836487 1-38654705669' 2020-03-11T22:49:51.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:51.876 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836487 2020-03-11T22:49:51.876 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:51.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:49:51.877 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836487 2020-03-11T22:49:51.878 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:51.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836487 2020-03-11T22:49:51.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2020-03-11T22:49:51.879 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:49:51.879 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836487 2020-03-11T22:49:52.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836485 -lt 21474836487 2020-03-11T22:49:52.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:49:53.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:49:53.228 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:49:53.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836487 -lt 21474836487 2020-03-11T22:49:53.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:49:53.578 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705669 2020-03-11T22:49:53.578 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:49:53.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:49:53.580 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705669 2020-03-11T22:49:53.580 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:49:53.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705669 2020-03-11T22:49:53.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705669' 2020-03-11T22:49:53.581 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 38654705669 2020-03-11T22:49:53.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:49:53.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705669 -lt 38654705669 2020-03-11T22:49:53.928 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:49:53.928 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:49:53.929 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:49:54.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 3 == 0 2020-03-11T22:49:54.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:49:54.405 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:49:54.406 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:49:54.406 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:49:54.406 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:49:54.406 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:49:54.406 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:49:54.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=3 2020-03-11T22:49:54.955 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:49:54.955 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:49:54.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:49:55.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 3 = 3 2020-03-11T22:49:55.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:49:55.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:49:55.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:348: TEST_backfill_pool_priority: ceph pg dump pgs 2020-03-11T22:49:55.532 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:49:55.532 INFO:tasks.workunit.client.0.smithi145.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:49:46.042672+0000 0'0 26:18 [1,0] 1 [1,0] 1 0'0 2020-03-11T22:49:44.186681+0000 0'0 2020-03-11T22:49:44.186681+0000 0 2020-03-11T22:49:55.532 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:49:41.874129+0000 0'0 26:28 [1,0] 1 [1,0] 1 0'0 2020-03-11T22:49:39.670019+0000 0'0 2020-03-11T22:49:39.670019+0000 0 2020-03-11T22:49:55.533 INFO:tasks.workunit.client.0.smithi145.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:49:44.659074+0000 0'0 26:24 [0,1] 0 [0,1] 0 0'0 2020-03-11T22:49:42.015190+0000 0'0 2020-03-11T22:49:42.015190+0000 0 2020-03-11T22:49:55.533 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:49:55.533 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:49:55.534 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:49:55.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:352: TEST_backfill_pool_priority: local PG1 2020-03-11T22:49:55.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:353: TEST_backfill_pool_priority: local POOLNUM1 2020-03-11T22:49:55.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:354: TEST_backfill_pool_priority: local pool1 2020-03-11T22:49:55.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:355: TEST_backfill_pool_priority: local chk_osd1_1 2020-03-11T22:49:55.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:356: TEST_backfill_pool_priority: local chk_osd1_2 2020-03-11T22:49:55.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:358: TEST_backfill_pool_priority: local PG2 2020-03-11T22:49:55.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:359: TEST_backfill_pool_priority: local POOLNUM2 2020-03-11T22:49:55.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:360: TEST_backfill_pool_priority: local pool2 2020-03-11T22:49:55.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:361: TEST_backfill_pool_priority: local chk_osd2_1 2020-03-11T22:49:55.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:362: TEST_backfill_pool_priority: local chk_osd2_2 2020-03-11T22:49:55.544 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:364: TEST_backfill_pool_priority: seq 1 3 2020-03-11T22:49:55.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:364: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:55.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: ceph pg map 1.0 --format=json 2020-03-11T22:49:55.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: jq '.acting[]' 2020-03-11T22:49:55.878 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:49:55.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: local test_osd1=1 2020-03-11T22:49:55.880 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:49:55.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: local test_osd2=0 2020-03-11T22:49:55.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:369: TEST_backfill_pool_priority: '[' -z '' ']' 2020-03-11T22:49:55.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:371: TEST_backfill_pool_priority: PG1=1.0 2020-03-11T22:49:55.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:372: TEST_backfill_pool_priority: POOLNUM1=1 2020-03-11T22:49:55.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:373: TEST_backfill_pool_priority: pool1=test1 2020-03-11T22:49:55.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:374: TEST_backfill_pool_priority: chk_osd1_1=1 2020-03-11T22:49:55.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:375: TEST_backfill_pool_priority: chk_osd1_2=0 2020-03-11T22:49:55.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:364: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:55.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: ceph pg map 2.0 --format=json 2020-03-11T22:49:55.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: jq '.acting[]' 2020-03-11T22:49:56.216 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:49:56.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: local test_osd1=0 2020-03-11T22:49:56.217 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:49:56.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: local test_osd2=1 2020-03-11T22:49:56.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:369: TEST_backfill_pool_priority: '[' -z 1.0 ']' 2020-03-11T22:49:56.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:376: TEST_backfill_pool_priority: '[' 1 '!=' 0 ']' 2020-03-11T22:49:56.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:378: TEST_backfill_pool_priority: PG2=2.0 2020-03-11T22:49:56.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:379: TEST_backfill_pool_priority: POOLNUM2=2 2020-03-11T22:49:56.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:380: TEST_backfill_pool_priority: pool2=test2 2020-03-11T22:49:56.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:381: TEST_backfill_pool_priority: chk_osd2_1=0 2020-03-11T22:49:56.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:382: TEST_backfill_pool_priority: chk_osd2_2=1 2020-03-11T22:49:56.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:383: TEST_backfill_pool_priority: break 2020-03-11T22:49:56.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:386: TEST_backfill_pool_priority: rm -f td/osd-backfill-prio/acting 2020-03-11T22:49:56.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:388: TEST_backfill_pool_priority: '[' test2 = '' ']' 2020-03-11T22:49:56.221 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: seq 1 3 2020-03-11T22:49:56.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:56.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:396: TEST_backfill_pool_priority: '[' 1 '!=' 1 -a 1 '!=' 2 ']' 2020-03-11T22:49:56.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:56.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:396: TEST_backfill_pool_priority: '[' 2 '!=' 1 -a 2 '!=' 2 ']' 2020-03-11T22:49:56.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:56.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:396: TEST_backfill_pool_priority: '[' 3 '!=' 1 -a 3 '!=' 2 ']' 2020-03-11T22:49:56.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:398: TEST_backfill_pool_priority: delete_pool test3 2020-03-11T22:49:56.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test3 2020-03-11T22:49:56.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2020-03-11T22:49:56.748 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test3' does not exist 2020-03-11T22:49:56.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:402: TEST_backfill_pool_priority: pool1_extra_prio=1 2020-03-11T22:49:56.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:403: TEST_backfill_pool_priority: pool2_extra_prio=2 2020-03-11T22:49:56.757 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:405: TEST_backfill_pool_priority: expr 150 + 1 + 1 2020-03-11T22:49:56.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:405: TEST_backfill_pool_priority: pool1_prio=152 2020-03-11T22:49:56.758 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:406: TEST_backfill_pool_priority: expr 150 + 1 + 2 2020-03-11T22:49:56.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:406: TEST_backfill_pool_priority: pool2_prio=153 2020-03-11T22:49:56.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:408: TEST_backfill_pool_priority: ceph osd pool set test1 size 1 2020-03-11T22:49:57.426 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 1 2020-03-11T22:49:57.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:409: TEST_backfill_pool_priority: ceph osd pool set test1 recovery_priority 1 2020-03-11T22:49:58.094 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 recovery_priority to 1 2020-03-11T22:49:58.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:410: TEST_backfill_pool_priority: ceph osd pool set test2 size 1 2020-03-11T22:49:58.862 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 2 size to 1 2020-03-11T22:49:58.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:411: TEST_backfill_pool_priority: ceph osd pool set test2 recovery_priority 2 2020-03-11T22:49:59.508 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 2 recovery_priority to 2 2020-03-11T22:49:59.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:412: TEST_backfill_pool_priority: wait_for_clean 2020-03-11T22:49:59.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:49:59.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:49:59.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:49:59.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:49:59.519 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:49:59.519 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:49:59.519 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:49:59.519 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:49:59.519 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:49:59.519 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:49:59.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:49:59.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:49:59.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:49:59.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:49:59.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:49:59.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:49:59.978 INFO:tasks.workunit.client.0.smithi145.stderr:1' 2020-03-11T22:49:59.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:49:59.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:49:59.979 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:50:00.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836491 2020-03-11T22:50:00.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836491' 2020-03-11T22:50:00.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:00.104 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:50:00.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705673 2020-03-11T22:50:00.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836491 1-38654705673' 2020-03-11T22:50:00.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:00.227 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836491 2020-03-11T22:50:00.227 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:00.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:50:00.229 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836491 2020-03-11T22:50:00.230 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:00.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836491 2020-03-11T22:50:00.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2020-03-11T22:50:00.231 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:50:00.231 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836491 2020-03-11T22:50:00.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836488 -lt 21474836491 2020-03-11T22:50:00.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:50:01.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:50:01.580 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:50:01.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836491 -lt 21474836491 2020-03-11T22:50:01.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:01.933 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705673 2020-03-11T22:50:01.933 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:01.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:50:01.934 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705673 2020-03-11T22:50:01.935 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:01.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705673 2020-03-11T22:50:01.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705673' 2020-03-11T22:50:01.936 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 38654705673 2020-03-11T22:50:01.936 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:50:02.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705673 -lt 38654705673 2020-03-11T22:50:02.304 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:50:02.304 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:02.305 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:02.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 2 == 0 2020-03-11T22:50:02.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:02.750 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:02.751 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:02.751 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:02.751 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:02.751 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:02.751 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:03.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:50:03.083 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:03.083 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:03.083 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:03.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 2 2020-03-11T22:50:03.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:50:03.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:50:03.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:414: TEST_backfill_pool_priority: dd if=/dev/urandom of=td/osd-backfill-prio/data bs=1M count=10 2020-03-11T22:50:03.575 INFO:tasks.workunit.client.0.smithi145.stderr:10+0 records in 2020-03-11T22:50:03.576 INFO:tasks.workunit.client.0.smithi145.stderr:10+0 records out 2020-03-11T22:50:03.576 INFO:tasks.workunit.client.0.smithi145.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0460135 s, 228 MB/s 2020-03-11T22:50:03.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:415: TEST_backfill_pool_priority: p=1 2020-03-11T22:50:03.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:416: TEST_backfill_pool_priority: for pname in $pool1 $pool2 2020-03-11T22:50:03.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: seq 1 50 2020-03-11T22:50:03.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:03.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj1-p1 td/osd-backfill-prio/data 2020-03-11T22:50:03.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:03.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj2-p1 td/osd-backfill-prio/data 2020-03-11T22:50:04.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:04.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj3-p1 td/osd-backfill-prio/data 2020-03-11T22:50:04.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:04.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj4-p1 td/osd-backfill-prio/data 2020-03-11T22:50:04.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:04.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj5-p1 td/osd-backfill-prio/data 2020-03-11T22:50:04.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:04.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj6-p1 td/osd-backfill-prio/data 2020-03-11T22:50:05.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:05.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj7-p1 td/osd-backfill-prio/data 2020-03-11T22:50:05.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:05.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj8-p1 td/osd-backfill-prio/data 2020-03-11T22:50:05.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:05.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj9-p1 td/osd-backfill-prio/data 2020-03-11T22:50:05.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:05.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj10-p1 td/osd-backfill-prio/data 2020-03-11T22:50:06.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:06.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj11-p1 td/osd-backfill-prio/data 2020-03-11T22:50:06.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:06.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj12-p1 td/osd-backfill-prio/data 2020-03-11T22:50:06.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:06.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj13-p1 td/osd-backfill-prio/data 2020-03-11T22:50:06.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:06.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj14-p1 td/osd-backfill-prio/data 2020-03-11T22:50:07.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:07.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj15-p1 td/osd-backfill-prio/data 2020-03-11T22:50:07.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:07.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj16-p1 td/osd-backfill-prio/data 2020-03-11T22:50:07.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:07.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj17-p1 td/osd-backfill-prio/data 2020-03-11T22:50:07.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:07.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj18-p1 td/osd-backfill-prio/data 2020-03-11T22:50:08.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:08.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj19-p1 td/osd-backfill-prio/data 2020-03-11T22:50:08.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:08.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj20-p1 td/osd-backfill-prio/data 2020-03-11T22:50:08.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:08.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj21-p1 td/osd-backfill-prio/data 2020-03-11T22:50:08.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:08.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj22-p1 td/osd-backfill-prio/data 2020-03-11T22:50:09.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:09.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj23-p1 td/osd-backfill-prio/data 2020-03-11T22:50:09.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:09.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj24-p1 td/osd-backfill-prio/data 2020-03-11T22:50:09.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:09.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj25-p1 td/osd-backfill-prio/data 2020-03-11T22:50:09.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:09.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj26-p1 td/osd-backfill-prio/data 2020-03-11T22:50:10.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:10.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj27-p1 td/osd-backfill-prio/data 2020-03-11T22:50:10.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:10.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj28-p1 td/osd-backfill-prio/data 2020-03-11T22:50:10.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:10.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj29-p1 td/osd-backfill-prio/data 2020-03-11T22:50:10.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:10.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj30-p1 td/osd-backfill-prio/data 2020-03-11T22:50:11.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:11.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj31-p1 td/osd-backfill-prio/data 2020-03-11T22:50:11.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:11.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj32-p1 td/osd-backfill-prio/data 2020-03-11T22:50:11.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:11.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj33-p1 td/osd-backfill-prio/data 2020-03-11T22:50:11.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:11.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj34-p1 td/osd-backfill-prio/data 2020-03-11T22:50:12.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:12.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj35-p1 td/osd-backfill-prio/data 2020-03-11T22:50:12.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:12.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj36-p1 td/osd-backfill-prio/data 2020-03-11T22:50:12.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:12.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj37-p1 td/osd-backfill-prio/data 2020-03-11T22:50:12.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:12.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj38-p1 td/osd-backfill-prio/data 2020-03-11T22:50:12.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:12.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj39-p1 td/osd-backfill-prio/data 2020-03-11T22:50:13.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:13.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj40-p1 td/osd-backfill-prio/data 2020-03-11T22:50:13.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:13.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj41-p1 td/osd-backfill-prio/data 2020-03-11T22:50:13.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:13.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj42-p1 td/osd-backfill-prio/data 2020-03-11T22:50:13.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:13.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj43-p1 td/osd-backfill-prio/data 2020-03-11T22:50:14.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:14.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj44-p1 td/osd-backfill-prio/data 2020-03-11T22:50:14.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:14.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj45-p1 td/osd-backfill-prio/data 2020-03-11T22:50:14.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:14.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj46-p1 td/osd-backfill-prio/data 2020-03-11T22:50:14.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:14.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj47-p1 td/osd-backfill-prio/data 2020-03-11T22:50:15.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:15.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj48-p1 td/osd-backfill-prio/data 2020-03-11T22:50:15.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:15.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj49-p1 td/osd-backfill-prio/data 2020-03-11T22:50:15.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:15.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj50-p1 td/osd-backfill-prio/data 2020-03-11T22:50:15.918 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: expr 1 + 1 2020-03-11T22:50:15.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: p=2 2020-03-11T22:50:15.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:416: TEST_backfill_pool_priority: for pname in $pool1 $pool2 2020-03-11T22:50:15.920 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: seq 1 50 2020-03-11T22:50:15.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:15.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj1-p2 td/osd-backfill-prio/data 2020-03-11T22:50:16.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:16.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj2-p2 td/osd-backfill-prio/data 2020-03-11T22:50:16.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:16.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj3-p2 td/osd-backfill-prio/data 2020-03-11T22:50:16.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:16.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj4-p2 td/osd-backfill-prio/data 2020-03-11T22:50:17.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:17.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj5-p2 td/osd-backfill-prio/data 2020-03-11T22:50:17.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:17.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj6-p2 td/osd-backfill-prio/data 2020-03-11T22:50:17.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:17.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj7-p2 td/osd-backfill-prio/data 2020-03-11T22:50:17.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:17.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj8-p2 td/osd-backfill-prio/data 2020-03-11T22:50:18.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:18.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj9-p2 td/osd-backfill-prio/data 2020-03-11T22:50:18.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:18.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj10-p2 td/osd-backfill-prio/data 2020-03-11T22:50:18.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:18.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj11-p2 td/osd-backfill-prio/data 2020-03-11T22:50:18.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:18.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj12-p2 td/osd-backfill-prio/data 2020-03-11T22:50:19.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:19.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj13-p2 td/osd-backfill-prio/data 2020-03-11T22:50:19.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:19.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj14-p2 td/osd-backfill-prio/data 2020-03-11T22:50:19.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:19.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj15-p2 td/osd-backfill-prio/data 2020-03-11T22:50:19.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:19.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj16-p2 td/osd-backfill-prio/data 2020-03-11T22:50:20.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:20.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj17-p2 td/osd-backfill-prio/data 2020-03-11T22:50:20.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:20.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj18-p2 td/osd-backfill-prio/data 2020-03-11T22:50:20.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:20.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj19-p2 td/osd-backfill-prio/data 2020-03-11T22:50:20.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:20.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj20-p2 td/osd-backfill-prio/data 2020-03-11T22:50:21.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:21.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj21-p2 td/osd-backfill-prio/data 2020-03-11T22:50:21.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:21.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj22-p2 td/osd-backfill-prio/data 2020-03-11T22:50:21.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:21.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj23-p2 td/osd-backfill-prio/data 2020-03-11T22:50:21.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:21.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj24-p2 td/osd-backfill-prio/data 2020-03-11T22:50:22.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:22.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj25-p2 td/osd-backfill-prio/data 2020-03-11T22:50:22.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:22.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj26-p2 td/osd-backfill-prio/data 2020-03-11T22:50:22.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:22.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj27-p2 td/osd-backfill-prio/data 2020-03-11T22:50:22.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:22.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj28-p2 td/osd-backfill-prio/data 2020-03-11T22:50:23.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:23.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj29-p2 td/osd-backfill-prio/data 2020-03-11T22:50:23.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:23.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj30-p2 td/osd-backfill-prio/data 2020-03-11T22:50:23.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:23.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj31-p2 td/osd-backfill-prio/data 2020-03-11T22:50:23.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:23.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj32-p2 td/osd-backfill-prio/data 2020-03-11T22:50:24.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:24.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj33-p2 td/osd-backfill-prio/data 2020-03-11T22:50:24.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:24.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj34-p2 td/osd-backfill-prio/data 2020-03-11T22:50:24.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:24.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj35-p2 td/osd-backfill-prio/data 2020-03-11T22:50:24.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:24.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj36-p2 td/osd-backfill-prio/data 2020-03-11T22:50:25.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:25.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj37-p2 td/osd-backfill-prio/data 2020-03-11T22:50:25.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:25.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj38-p2 td/osd-backfill-prio/data 2020-03-11T22:50:25.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:25.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj39-p2 td/osd-backfill-prio/data 2020-03-11T22:50:25.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:25.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj40-p2 td/osd-backfill-prio/data 2020-03-11T22:50:26.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:26.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj41-p2 td/osd-backfill-prio/data 2020-03-11T22:50:26.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:26.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj42-p2 td/osd-backfill-prio/data 2020-03-11T22:50:26.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:26.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj43-p2 td/osd-backfill-prio/data 2020-03-11T22:50:26.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:26.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj44-p2 td/osd-backfill-prio/data 2020-03-11T22:50:27.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:27.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj45-p2 td/osd-backfill-prio/data 2020-03-11T22:50:27.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:27.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj46-p2 td/osd-backfill-prio/data 2020-03-11T22:50:27.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:27.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj47-p2 td/osd-backfill-prio/data 2020-03-11T22:50:27.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:27.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj48-p2 td/osd-backfill-prio/data 2020-03-11T22:50:28.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:28.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj49-p2 td/osd-backfill-prio/data 2020-03-11T22:50:28.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:28.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj50-p2 td/osd-backfill-prio/data 2020-03-11T22:50:28.644 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: expr 2 + 1 2020-03-11T22:50:28.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: p=3 2020-03-11T22:50:28.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:425: TEST_backfill_pool_priority: get_not_primary test1 obj1-p1 2020-03-11T22:50:28.647 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1166: get_not_primary: local poolname=test1 2020-03-11T22:50:28.647 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local objectname=obj1-p1 2020-03-11T22:50:28.647 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: get_primary test1 obj1-p1 2020-03-11T22:50:28.647 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: get_primary: local poolname=test1 2020-03-11T22:50:28.647 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local objectname=obj1-p1 2020-03-11T22:50:28.647 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: ceph --format json osd map test1 obj1-p1 2020-03-11T22:50:28.647 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: jq .acting_primary 2020-03-11T22:50:29.003 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: local primary=1 2020-03-11T22:50:29.003 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: ceph --format json osd map test1 obj1-p1 2020-03-11T22:50:29.003 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2020-03-11T22:50:29.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:425: TEST_backfill_pool_priority: local otherosd=null 2020-03-11T22:50:29.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:427: TEST_backfill_pool_priority: ceph pg dump pgs 2020-03-11T22:50:29.680 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:50:29.680 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:49:57.582241+0000 38'150 38:196 [1] 1 [1] 1 0'0 2020-03-11T22:49:39.670019+0000 0'0 2020-03-11T22:49:39.670019+0000 0 2020-03-11T22:50:29.680 INFO:tasks.workunit.client.0.smithi145.stdout:2.0 30 0 0 0 0 314572800 0 0 90 90 active+clean 2020-03-11T22:49:58.745340+0000 38'90 38:129 [0] 0 [0] 0 0'0 2020-03-11T22:49:42.015190+0000 0'0 2020-03-11T22:49:42.015190+0000 0 2020-03-11T22:50:29.680 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:50:29.680 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:50:29.681 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:50:29.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:428: TEST_backfill_pool_priority: ERRORS=0 2020-03-11T22:50:29.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:430: TEST_backfill_pool_priority: ceph osd pool set test1 size 2 2020-03-11T22:50:30.396 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 2 2020-03-11T22:50:30.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:431: TEST_backfill_pool_priority: ceph osd pool set test2 size 2 2020-03-11T22:50:31.432 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 2 size to 2 2020-03-11T22:50:31.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:432: TEST_backfill_pool_priority: sleep 5 2020-03-11T22:50:36.444 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:433: TEST_backfill_pool_priority: get_asok_path osd.1 2020-03-11T22:50:36.444 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:50:36.444 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:50:36.444 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:50:36.444 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:50:36.444 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56096 2020-03-11T22:50:36.445 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56096/ceph-osd.1.asok 2020-03-11T22:50:36.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:433: TEST_backfill_pool_priority: CEPH_ARGS= 2020-03-11T22:50:36.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:433: TEST_backfill_pool_priority: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:50:36.550 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 2020-03-11T22:50:36.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:434: TEST_backfill_pool_priority: echo osd.1 2020-03-11T22:50:36.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:435: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-03-11T22:50:36.551 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:50:36.551 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:50:36.551 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:50:36.551 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:50:36.551 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:50:36.551 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:50:36.551 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:50:36.552 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:50:36.552 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 152, 2020-03-11T22:50:36.552 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:50:36.552 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:50:36.552 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:50:36.552 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:50:36.552 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:50:36.552 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:50:36.553 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:50:36.553 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:50:36.553 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:50:36.554 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:50:36.554 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "2.0", 2020-03-11T22:50:36.554 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 153, 2020-03-11T22:50:36.554 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:50:36.554 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:50:36.554 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:50:36.555 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:50:36.555 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:50:36.556 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:436: TEST_backfill_pool_priority: get_asok_path osd.0 2020-03-11T22:50:36.556 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:50:36.556 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:50:36.556 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:50:36.556 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:50:36.556 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56096 2020-03-11T22:50:36.557 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56096/ceph-osd.0.asok 2020-03-11T22:50:36.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:436: TEST_backfill_pool_priority: CEPH_ARGS= 2020-03-11T22:50:36.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:436: TEST_backfill_pool_priority: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.0.asok dump_recovery_reservations 2020-03-11T22:50:36.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:437: TEST_backfill_pool_priority: echo osd.0 2020-03-11T22:50:36.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:438: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-03-11T22:50:36.649 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 2020-03-11T22:50:36.650 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:50:36.650 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:50:36.650 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:50:36.650 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:50:36.650 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:50:36.650 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:50:36.650 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:50:36.651 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "2.0", 2020-03-11T22:50:36.651 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 153, 2020-03-11T22:50:36.651 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:50:36.651 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:50:36.651 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:50:36.651 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:50:36.651 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:50:36.651 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:50:36.652 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:50:36.652 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:50:36.652 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:50:36.652 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:50:36.652 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:50:36.652 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 152, 2020-03-11T22:50:36.652 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:50:36.653 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:50:36.653 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:50:36.653 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:50:36.653 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:50:36.654 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-03-11T22:50:36.654 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].item' 2020-03-11T22:50:36.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: eval 'ITEM="1.0"' 2020-03-11T22:50:36.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: ITEM=1.0 2020-03-11T22:50:36.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:442: TEST_backfill_pool_priority: '[' 1.0 '!=' 1.0 ']' 2020-03-11T22:50:36.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:447: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-03-11T22:50:36.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:447: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].prio' 2020-03-11T22:50:36.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:447: TEST_backfill_pool_priority: PRIO=152 2020-03-11T22:50:36.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:448: TEST_backfill_pool_priority: '[' 152 '!=' 152 ']' 2020-03-11T22:50:36.657 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-03-11T22:50:36.657 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].item' 2020-03-11T22:50:36.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: eval 'ITEM="1.0"' 2020-03-11T22:50:36.659 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: ITEM=1.0 2020-03-11T22:50:36.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:457: TEST_backfill_pool_priority: '[' 1.0 '!=' 1.0 ']' 2020-03-11T22:50:36.660 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:462: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-03-11T22:50:36.660 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:462: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2020-03-11T22:50:36.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:462: TEST_backfill_pool_priority: PRIO=152 2020-03-11T22:50:36.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:463: TEST_backfill_pool_priority: '[' 152 '!=' 152 ']' 2020-03-11T22:50:36.663 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-03-11T22:50:36.663 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].item' 2020-03-11T22:50:36.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: eval 'ITEM="2.0"' 2020-03-11T22:50:36.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: ITEM=2.0 2020-03-11T22:50:36.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:472: TEST_backfill_pool_priority: '[' 2.0 '!=' 2.0 ']' 2020-03-11T22:50:36.667 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:477: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-03-11T22:50:36.667 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:477: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].prio' 2020-03-11T22:50:36.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:477: TEST_backfill_pool_priority: PRIO=153 2020-03-11T22:50:36.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:478: TEST_backfill_pool_priority: '[' 153 '!=' 153 ']' 2020-03-11T22:50:36.670 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-03-11T22:50:36.670 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].item' 2020-03-11T22:50:36.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: eval 'ITEM="2.0"' 2020-03-11T22:50:36.672 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: ITEM=2.0 2020-03-11T22:50:36.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:487: TEST_backfill_pool_priority: '[' 2.0 '!=' 2.0 ']' 2020-03-11T22:50:36.673 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:492: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-03-11T22:50:36.673 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:492: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2020-03-11T22:50:36.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:492: TEST_backfill_pool_priority: PRIO=153 2020-03-11T22:50:36.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:493: TEST_backfill_pool_priority: '[' 153 '!=' 153 ']' 2020-03-11T22:50:36.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:500: TEST_backfill_pool_priority: wait_for_clean 2020-03-11T22:50:36.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:50:36.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:50:36.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:50:36.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:50:36.678 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:50:36.678 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:50:36.678 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:50:36.678 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:50:36.678 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:50:36.678 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:50:36.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:50:36.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:50:36.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:50:36.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:50:36.786 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:50:37.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:50:37.122 INFO:tasks.workunit.client.0.smithi145.stderr:1' 2020-03-11T22:50:37.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:50:37.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:37.123 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:50:37.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836500 2020-03-11T22:50:37.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836500' 2020-03-11T22:50:37.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:37.252 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:50:37.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705682 2020-03-11T22:50:37.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836500 1-38654705682' 2020-03-11T22:50:37.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:37.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836500 2020-03-11T22:50:37.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:37.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:50:37.404 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836500 2020-03-11T22:50:37.405 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:37.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836500 2020-03-11T22:50:37.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2020-03-11T22:50:37.406 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836500 2020-03-11T22:50:37.406 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:50:37.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836498 -lt 21474836500 2020-03-11T22:50:37.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:50:38.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:50:38.792 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:50:39.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836500 -lt 21474836500 2020-03-11T22:50:39.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:39.262 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705682 2020-03-11T22:50:39.262 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:39.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:50:39.264 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705682 2020-03-11T22:50:39.264 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:39.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705682 2020-03-11T22:50:39.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705682' 2020-03-11T22:50:39.265 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 38654705682 2020-03-11T22:50:39.266 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:50:39.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705682 -lt 38654705682 2020-03-11T22:50:39.844 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:50:39.844 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:39.845 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:40.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 2 == 0 2020-03-11T22:50:40.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:40.433 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:40.433 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:40.434 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:40.434 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:40.434 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:40.434 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:40.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:50:40.776 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:40.776 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:40.777 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:41.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:50:41.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:50:41.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:50:41.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:50:41.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:50:41.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:41.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:41.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:41.405 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:41.405 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:41.405 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:41.405 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:41.406 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:41.406 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:41.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:50:41.767 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:41.767 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:41.767 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:42.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:50:42.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:50:42.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:42.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:42.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:42.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:42.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:42.227 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:42.227 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:42.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=29966495 2020-03-11T22:50:42.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 29966495 '!=' null 2020-03-11T22:50:42.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:42.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:50:42.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:42.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:42.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:42.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:42.896 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:42.896 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:42.896 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:42.896 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:42.896 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:43.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:50:43.233 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:43.233 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:43.234 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:43.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:50:43.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:50:43.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:43.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:43.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:43.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:43.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:43.683 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:43.683 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:44.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=25230625 2020-03-11T22:50:44.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 25230625 '!=' null 2020-03-11T22:50:44.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:44.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:50:44.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:44.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:44.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:44.340 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:44.340 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:44.341 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:44.341 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:44.341 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:44.341 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:44.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:50:44.731 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:44.731 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:44.731 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:45.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:50:45.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:50:45.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:45.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:45.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:45.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:45.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:45.208 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:45.208 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:45.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=29703222 2020-03-11T22:50:45.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 29703222 '!=' null 2020-03-11T22:50:45.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:45.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:50:45.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:45.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:45.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:45.961 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:45.961 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:45.961 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:45.961 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:45.962 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:45.962 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:46.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:50:46.331 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:46.332 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:46.332 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:46.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:50:46.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:50:46.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:46.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:46.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:46.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:46.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:46.854 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:46.854 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:47.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=34945462 2020-03-11T22:50:47.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 34945462 '!=' null 2020-03-11T22:50:47.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:47.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:50:47.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:47.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:47.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:47.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:47.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:47.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:47.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:47.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:47.404 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:47.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:50:47.742 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:47.742 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:47.742 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:48.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:50:48.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:50:48.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:48.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:48.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:48.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:48.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:48.194 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:48.194 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:48.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=34945462 2020-03-11T22:50:48.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 34945462 '!=' null 2020-03-11T22:50:48.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:48.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:50:48.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:48.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:48.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:48.886 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:48.887 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:48.887 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:48.887 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:48.887 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:48.887 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:49.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:50:49.251 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:49.252 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:49.252 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:49.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:50:49.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:50:49.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:49.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:49.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:49.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:49.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:49.707 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:49.707 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:50.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=28830319 2020-03-11T22:50:50.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 28830319 '!=' null 2020-03-11T22:50:50.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:50.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:50:50.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:50.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:50.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:50.277 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:50.277 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:50.277 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:50.278 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:50.278 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:50.278 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:50.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:50:50.759 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:50.759 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:50.760 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:51.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:50:51.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:50:51.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:51.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:51.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:51.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:51.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:51.214 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:51.214 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:51.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=34946243 2020-03-11T22:50:51.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 34946243 '!=' null 2020-03-11T22:50:51.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:51.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:50:51.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:51.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:51.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:51.769 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:51.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:51.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:51.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:51.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:51.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:52.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:50:52.116 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:52.117 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:52.117 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:52.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:50:52.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:50:52.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:52.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:52.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:52.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:52.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:52.710 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:52.711 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:53.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=29704911 2020-03-11T22:50:53.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 29704911 '!=' null 2020-03-11T22:50:53.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:53.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:50:53.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:53.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:53.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:53.415 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:53.415 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:53.415 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:53.415 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:53.415 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:53.416 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:53.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:50:53.762 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:53.762 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:53.763 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:54.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:50:54.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:50:54.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:54.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:54.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:54.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:54.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:54.217 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:54.217 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:54.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=29704911 2020-03-11T22:50:54.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 29704911 '!=' null 2020-03-11T22:50:54.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:54.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:50:54.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:54.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:54.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:54.765 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:54.765 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:54.765 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:54.765 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:54.766 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:54.766 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:55.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:50:55.103 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:55.104 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:55.104 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:55.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:50:55.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:50:55.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:55.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:55.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:55.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:55.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:55.613 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:55.613 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:56.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=37567673 2020-03-11T22:50:56.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 37567673 '!=' null 2020-03-11T22:50:56.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:56.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:50:56.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:56.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:56.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:56.164 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:56.164 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:56.164 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:56.165 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:56.165 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:56.165 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:56.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:50:56.500 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:56.501 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:56.501 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:57.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:50:57.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:50:57.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:57.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:57.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:57.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:57.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:57.196 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:57.196 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:57.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=38441266 2020-03-11T22:50:57.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 38441266 '!=' null 2020-03-11T22:50:57.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:57.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:50:57.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:58.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:58.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:58.089 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:58.090 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:58.090 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:58.090 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:58.090 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:58.090 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:58.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:50:58.469 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:58.469 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:58.470 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:58.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:50:58.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:50:58.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:50:58.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:50:58.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:50:58.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:50:58.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:50:58.921 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:50:58.921 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:50:59.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=30578136 2020-03-11T22:50:59.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 30578136 '!=' null 2020-03-11T22:50:59.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:50:59.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:50:59.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:50:59.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:50:59.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:59.674 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:59.674 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:59.675 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:59.675 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:59.675 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:59.675 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:00.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:00.064 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:00.064 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:00.064 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:00.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:51:00.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:00.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:00.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:00.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:00.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:00.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:00.606 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:00.606 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:01.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=36693381 2020-03-11T22:51:01.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 36693381 '!=' null 2020-03-11T22:51:01.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:01.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:51:01.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:01.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:01.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:01.185 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:01.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:01.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:01.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:01.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:01.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:01.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:01.540 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:01.540 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:01.540 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:01.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:51:01.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:01.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:01.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:01.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:01.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:01.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:01.991 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:01.991 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:02.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=36693381 2020-03-11T22:51:02.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 36693381 '!=' null 2020-03-11T22:51:02.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:02.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:51:02.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:02.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:02.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:02.558 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:02.558 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:02.558 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:02.558 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:02.558 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:02.558 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:02.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:02.903 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:02.904 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:02.904 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:03.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:51:03.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:03.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:03.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:03.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:03.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:03.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:03.347 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:03.347 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:03.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=44555993 2020-03-11T22:51:03.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 44555993 '!=' null 2020-03-11T22:51:03.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:03.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:51:03.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:03.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:03.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:03.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:03.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:03.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:03.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:03.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:03.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:04.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:04.226 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:04.226 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:04.226 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:04.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:51:04.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:04.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:04.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:04.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:04.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:04.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:04.678 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:04.678 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:05.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=12568244 2020-03-11T22:51:05.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 12568244 '!=' null 2020-03-11T22:51:05.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:05.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:51:05.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:05.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:05.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:05.230 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:05.230 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:05.231 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:05.231 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:05.231 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:05.231 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:05.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:51:05.563 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:05.563 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:05.563 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:06.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 2 2020-03-11T22:51:06.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 0 2020-03-11T22:51:06.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:51:06.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=1 2020-03-11T22:51:06.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:51:06.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:06.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:06.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:06.106 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:06.107 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:06.107 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:06.107 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:06.107 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:06.107 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:06.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:51:06.438 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:06.438 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:06.438 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:06.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 2 2020-03-11T22:51:06.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:51:06.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:06.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:06.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:06.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:06.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:06.886 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:06.887 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:07.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=8736317 2020-03-11T22:51:07.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 8736317 '!=' null 2020-03-11T22:51:07.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:07.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:51:07.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:07.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:07.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:07.457 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:07.457 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:07.457 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:07.458 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:07.458 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:07.458 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:07.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:51:07.791 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:07.791 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:07.792 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:08.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 2 2020-03-11T22:51:08.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:51:08.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:51:08.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:502: TEST_backfill_pool_priority: '[' 0 '!=' 0 ']' 2020-03-11T22:51:08.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:506: TEST_backfill_pool_priority: echo TEST PASSED 2020-03-11T22:51:08.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:509: TEST_backfill_pool_priority: delete_pool test1 2020-03-11T22:51:08.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test1 2020-03-11T22:51:08.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2020-03-11T22:51:08.240 INFO:tasks.workunit.client.0.smithi145.stdout:TEST PASSED 2020-03-11T22:51:08.741 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test1' does not exist 2020-03-11T22:51:08.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:510: TEST_backfill_pool_priority: delete_pool test2 2020-03-11T22:51:08.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test2 2020-03-11T22:51:08.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2020-03-11T22:51:09.234 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test2' does not exist 2020-03-11T22:51:09.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:511: TEST_backfill_pool_priority: kill_daemons td/osd-backfill-prio 2020-03-11T22:51:09.243 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:51:09.243 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:51:09.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:51:09.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:51:09.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:51:09.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:51:09.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:512: TEST_backfill_pool_priority: return 0 2020-03-11T22:51:09.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:40: run: teardown td/osd-backfill-prio 2020-03-11T22:51:09.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-03-11T22:51:09.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:51:09.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-03-11T22:51:09.352 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:51:09.352 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:51:09.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:51:09.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:51:09.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:51:09.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:51:09.355 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:51:09.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:51:09.356 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:51:09.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:51:09.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:51:09.357 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:51:09.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:51:09.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:51:09.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:51:09.359 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:51:09.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:51:09.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:51:09.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-03-11T22:51:09.372 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:51:09.372 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:09.373 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56096 2020-03-11T22:51:09.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.56096 2020-03-11T22:51:09.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:51:09.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:51:09.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:37: run: for func in $funcs 2020-03-11T22:51:09.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:38: run: setup td/osd-backfill-prio 2020-03-11T22:51:09.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-prio 2020-03-11T22:51:09.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-prio 2020-03-11T22:51:09.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-03-11T22:51:09.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:51:09.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-03-11T22:51:09.376 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:51:09.376 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:51:09.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:51:09.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:51:09.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:51:09.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:51:09.377 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:51:09.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:51:09.378 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:51:09.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:51:09.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:51:09.378 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:51:09.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:51:09.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:51:09.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:51:09.380 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:51:09.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:51:09.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:51:09.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-03-11T22:51:09.382 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:51:09.382 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:09.382 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56096 2020-03-11T22:51:09.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.56096 2020-03-11T22:51:09.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:51:09.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:51:09.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-prio 2020-03-11T22:51:09.384 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:51:09.384 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:09.385 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56096 2020-03-11T22:51:09.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.56096 2020-03-11T22:51:09.386 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:51:09.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-03-11T22:51:09.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:51:09.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-prio 1' TERM HUP INT 2020-03-11T22:51:09.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:39: run: TEST_backfill_priority td/osd-backfill-prio 2020-03-11T22:51:09.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:46: TEST_backfill_priority: local dir=td/osd-backfill-prio 2020-03-11T22:51:09.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:47: TEST_backfill_priority: local pools=10 2020-03-11T22:51:09.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:48: TEST_backfill_priority: local OSDS=5 2020-03-11T22:51:09.387 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:50: TEST_backfill_priority: expr 150 + 1 2020-03-11T22:51:09.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:50: TEST_backfill_priority: local degraded_prio=151 2020-03-11T22:51:09.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:51: TEST_backfill_priority: local max_tries=10 2020-03-11T22:51:09.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:53: TEST_backfill_priority: run_mon td/osd-backfill-prio a 2020-03-11T22:51:09.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-prio 2020-03-11T22:51:09.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:51:09.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:51:09.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:51:09.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-prio/a 2020-03-11T22:51:09.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-prio/a --run-dir=td/osd-backfill-prio 2020-03-11T22:51:09.674 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:51:09.674 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:51:09.675 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:51:09.675 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:51:09.675 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:09.675 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56096 2020-03-11T22:51:09.675 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56096/$cluster-$name.asok' 2020-03-11T22:51:09.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-prio/a '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.56096/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-prio/log --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:51:09.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:51:09.819 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:51:09.819 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:51:09.819 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:51:09.819 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:51:09.819 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:51:09.819 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:51:09.820 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:51:09.820 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:51:09.820 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:51:09.820 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:09.820 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56096 2020-03-11T22:51:09.820 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56096/ceph-mon.a.asok 2020-03-11T22:51:09.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:51:09.821 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.56096/ceph-mon.a.asok config get fsid 2020-03-11T22:51:09.906 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:51:09.907 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:51:09.907 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:51:09.907 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:51:09.907 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:51:09.907 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:51:09.907 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:51:09.907 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:51:09.908 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:51:09.908 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:09.908 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56096 2020-03-11T22:51:09.908 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56096/ceph-mon.a.asok 2020-03-11T22:51:09.908 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:51:09.908 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.56096/ceph-mon.a.asok config get mon_host 2020-03-11T22:51:09.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:54: TEST_backfill_priority: run_mgr td/osd-backfill-prio x 2020-03-11T22:51:09.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-backfill-prio 2020-03-11T22:51:09.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:51:09.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:51:09.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:51:09.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-backfill-prio/x 2020-03-11T22:51:09.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:51:10.866 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:51:10.867 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:51:10.867 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:51:10.867 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:51:10.867 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:10.867 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56096 2020-03-11T22:51:10.867 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56096/$cluster-$name.asok' 2020-03-11T22:51:10.868 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:51:10.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-prio/x '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.56096/$cluster-$name.asok' --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:51:10.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:55: TEST_backfill_priority: export CEPH_ARGS 2020-03-11T22:51:10.890 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: expr 5 - 1 2020-03-11T22:51:10.892 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: seq 0 4 2020-03-11T22:51:10.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:51:10.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 0 2020-03-11T22:51:10.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:51:10.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:51:10.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:51:10.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:51:10.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/0 2020-03-11T22:51:10.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=9ff197a8-fd34-42fc-bffc-e43264b5b9b7 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:51:10.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:51:10.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:51:10.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:51:10.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/0' 2020-03-11T22:51:10.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/0/journal' 2020-03-11T22:51:10.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:51:10.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:51:10.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:51:10.898 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:51:10.898 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:51:10.898 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:51:10.898 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:51:10.898 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:10.899 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56096 2020-03-11T22:51:10.899 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56096/$cluster-$name.asok' 2020-03-11T22:51:10.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56096/$cluster-$name.asok' 2020-03-11T22:51:10.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:51:10.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:51:10.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:51:10.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:51:10.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:51:10.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:51:10.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:51:10.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:51:10.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:51:10.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:51:10.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/0 2020-03-11T22:51:10.900 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:51:10.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=b414eed6-e10a-4133-969e-7e2c07e346cf 2020-03-11T22:51:10.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 b414eed6-e10a-4133-969e-7e2c07e346cf' 2020-03-11T22:51:10.901 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 b414eed6-e10a-4133-969e-7e2c07e346cf 2020-03-11T22:51:10.901 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:51:10.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBea2ledkowNhAANjEICEez975nGBFRB62hnw== 2020-03-11T22:51:10.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBea2ledkowNhAANjEICEez975nGBFRB62hnw=="}' 2020-03-11T22:51:10.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new b414eed6-e10a-4133-969e-7e2c07e346cf -i td/osd-backfill-prio/0/new.json 2020-03-11T22:51:11.241 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:51:11.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/0/new.json 2020-03-11T22:51:11.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=9ff197a8-fd34-42fc-bffc-e43264b5b9b7 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56096/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBea2ledkowNhAANjEICEez975nGBFRB62hnw== --osd-uuid b414eed6-e10a-4133-969e-7e2c07e346cf 2020-03-11T22:51:11.277 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:11.274+0000 7f5bc363fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:11.279 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:11.276+0000 7f5bc363fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:11.280 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:11.277+0000 7f5bc363fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:11.280 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:11.277+0000 7f5bc363fec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:51:11.281 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:11.277+0000 7f5bc363fec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:51:11.281 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:11.277+0000 7f5bc363fec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:51:11.316 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:11.313+0000 7f5bc363fec0 -1 bluestore(td/osd-backfill-prio/0) _read_fsid unparsable uuid 2020-03-11T22:51:14.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/0/keyring 2020-03-11T22:51:14.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:51:14.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:51:14.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:51:14.295 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2020-03-11T22:51:14.762 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2020-03-11T22:51:14.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:51:14.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=9ff197a8-fd34-42fc-bffc-e43264b5b9b7 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56096/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:51:14.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:51:14.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:51:14.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:51:14.786 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:14.783+0000 7fce4af9aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:14.788 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:14.785+0000 7fce4af9aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:14.789 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:14.786+0000 7fce4af9aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:15.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:51:15.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:51:15.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:51:15.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:51:15.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:51:15.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:51:15.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:51:15.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:51:15.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:51:15.111 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:51:15.304 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:15.301+0000 7fce4af9aec0 -1 Falling back to public interface 2020-03-11T22:51:15.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:51:16.015 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:16.013+0000 7fce4af9aec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:51:16.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:51:16.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:51:16.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:51:16.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:51:16.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:51:16.475 INFO:tasks.workunit.client.0.smithi145.stdout:1 2020-03-11T22:51:16.831 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/63009,v1:127.0.0.1:6803/63009] [v2:127.0.0.1:6804/63009,v1:127.0.0.1:6805/63009] exists,up b414eed6-e10a-4133-969e-7e2c07e346cf 2020-03-11T22:51:16.832 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 48619ef5-1755-4dd7-9ed2-7690b5a3eaa8 2020-03-11T22:51:16.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:51:16.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:51:16.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:51:16.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:51:16.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 1 2020-03-11T22:51:16.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:51:16.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:51:16.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:51:16.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:51:16.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/1 2020-03-11T22:51:16.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=9ff197a8-fd34-42fc-bffc-e43264b5b9b7 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:51:16.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:51:16.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:51:16.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:51:16.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/1' 2020-03-11T22:51:16.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/1/journal' 2020-03-11T22:51:16.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:51:16.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:51:16.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:51:16.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:51:16.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:51:16.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:51:16.836 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:51:16.836 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:16.836 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56096 2020-03-11T22:51:16.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56096/$cluster-$name.asok' 2020-03-11T22:51:16.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56096/$cluster-$name.asok' 2020-03-11T22:51:16.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:51:16.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:51:16.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:51:16.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:51:16.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:51:16.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:51:16.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:51:16.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:51:16.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:51:16.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:51:16.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/1 2020-03-11T22:51:16.840 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:51:16.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=48619ef5-1755-4dd7-9ed2-7690b5a3eaa8 2020-03-11T22:51:16.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 48619ef5-1755-4dd7-9ed2-7690b5a3eaa8' 2020-03-11T22:51:16.840 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:51:16.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBka2leI/BOMhAA6TpPpGYNhS69yYGwB1DdUQ== 2020-03-11T22:51:16.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBka2leI/BOMhAA6TpPpGYNhS69yYGwB1DdUQ=="}' 2020-03-11T22:51:16.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 48619ef5-1755-4dd7-9ed2-7690b5a3eaa8 -i td/osd-backfill-prio/1/new.json 2020-03-11T22:51:17.346 INFO:tasks.workunit.client.0.smithi145.stdout:1 2020-03-11T22:51:17.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/1/new.json 2020-03-11T22:51:17.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=9ff197a8-fd34-42fc-bffc-e43264b5b9b7 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56096/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBka2leI/BOMhAA6TpPpGYNhS69yYGwB1DdUQ== --osd-uuid 48619ef5-1755-4dd7-9ed2-7690b5a3eaa8 2020-03-11T22:51:17.382 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:17.379+0000 7f6d372c1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:17.397 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:17.394+0000 7f6d372c1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:17.398 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:17.395+0000 7f6d372c1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:17.399 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:17.396+0000 7f6d372c1ec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:51:17.399 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:17.396+0000 7f6d372c1ec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:51:17.399 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:17.396+0000 7f6d372c1ec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:51:17.472 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:17.469+0000 7f6d372c1ec0 -1 bluestore(td/osd-backfill-prio/1) _read_fsid unparsable uuid 2020-03-11T22:51:20.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/1/keyring 2020-03-11T22:51:20.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:51:20.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:51:20.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:51:20.457 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2020-03-11T22:51:20.917 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2020-03-11T22:51:20.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:51:20.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=9ff197a8-fd34-42fc-bffc-e43264b5b9b7 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56096/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:51:20.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:51:20.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:51:20.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:51:20.941 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:20.938+0000 7f2e57b06ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:20.942 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:20.940+0000 7f2e57b06ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:20.944 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:20.941+0000 7f2e57b06ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:21.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:51:21.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:51:21.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:51:21.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:51:21.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:51:21.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:51:21.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:51:21.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:51:21.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:51:21.267 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:51:21.460 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:21.457+0000 7f2e57b06ec0 -1 Falling back to public interface 2020-03-11T22:51:21.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:51:22.187 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:22.184+0000 7f2e57b06ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:51:22.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:51:22.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:51:22.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:51:22.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:51:22.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:51:22.614 INFO:tasks.workunit.client.0.smithi145.stdout:1 2020-03-11T22:51:22.981 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/63364,v1:127.0.0.1:6811/63364] [v2:127.0.0.1:6812/63364,v1:127.0.0.1:6813/63364] exists,up 48619ef5-1755-4dd7-9ed2-7690b5a3eaa8 2020-03-11T22:51:22.981 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 718bb608-6aa5-4b8d-8b2e-933157f1a615 2020-03-11T22:51:22.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:51:22.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:51:22.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:51:22.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:51:22.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 2 2020-03-11T22:51:22.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:51:22.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:51:22.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:51:22.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:51:22.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/2 2020-03-11T22:51:22.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=9ff197a8-fd34-42fc-bffc-e43264b5b9b7 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:51:22.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:51:22.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:51:22.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:51:22.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/2' 2020-03-11T22:51:22.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/2/journal' 2020-03-11T22:51:22.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:51:22.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:51:22.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:51:22.984 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:51:22.984 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:51:22.984 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:51:22.984 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:51:22.985 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:22.985 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56096 2020-03-11T22:51:22.985 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56096/$cluster-$name.asok' 2020-03-11T22:51:22.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56096/$cluster-$name.asok' 2020-03-11T22:51:22.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:51:22.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:51:22.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:51:22.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:51:22.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:51:22.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:51:22.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:51:22.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:51:22.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:51:22.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:51:22.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/2 2020-03-11T22:51:22.987 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:51:22.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=718bb608-6aa5-4b8d-8b2e-933157f1a615 2020-03-11T22:51:22.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 718bb608-6aa5-4b8d-8b2e-933157f1a615' 2020-03-11T22:51:22.988 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:51:22.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBqa2leJtRHOxAAmsLcW98xNWG3/LvqogvoQw== 2020-03-11T22:51:22.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBqa2leJtRHOxAAmsLcW98xNWG3/LvqogvoQw=="}' 2020-03-11T22:51:22.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 718bb608-6aa5-4b8d-8b2e-933157f1a615 -i td/osd-backfill-prio/2/new.json 2020-03-11T22:51:23.556 INFO:tasks.workunit.client.0.smithi145.stdout:2 2020-03-11T22:51:23.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/2/new.json 2020-03-11T22:51:23.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=9ff197a8-fd34-42fc-bffc-e43264b5b9b7 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/2 --osd-journal=td/osd-backfill-prio/2/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56096/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBqa2leJtRHOxAAmsLcW98xNWG3/LvqogvoQw== --osd-uuid 718bb608-6aa5-4b8d-8b2e-933157f1a615 2020-03-11T22:51:23.592 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:23.589+0000 7fcac668fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:23.593 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:23.590+0000 7fcac668fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:23.595 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:23.592+0000 7fcac668fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:23.595 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:23.592+0000 7fcac668fec0 -1 bluestore(td/osd-backfill-prio/2/block) _read_bdev_label failed to open td/osd-backfill-prio/2/block: (2) No such file or directory 2020-03-11T22:51:23.595 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:23.592+0000 7fcac668fec0 -1 bluestore(td/osd-backfill-prio/2/block) _read_bdev_label failed to open td/osd-backfill-prio/2/block: (2) No such file or directory 2020-03-11T22:51:23.596 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:23.592+0000 7fcac668fec0 -1 bluestore(td/osd-backfill-prio/2/block) _read_bdev_label failed to open td/osd-backfill-prio/2/block: (2) No such file or directory 2020-03-11T22:51:23.677 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:23.674+0000 7fcac668fec0 -1 bluestore(td/osd-backfill-prio/2) _read_fsid unparsable uuid 2020-03-11T22:51:26.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/2/keyring 2020-03-11T22:51:26.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:51:26.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:51:26.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:51:26.779 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2020-03-11T22:51:27.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:51:27.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=9ff197a8-fd34-42fc-bffc-e43264b5b9b7 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/2 --osd-journal=td/osd-backfill-prio/2/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56096/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:51:27.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:51:27.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:51:27.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:51:27.236 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2020-03-11T22:51:27.259 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:27.256+0000 7fefa4ef4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:27.260 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:27.258+0000 7fefa4ef4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:27.262 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:27.260+0000 7fefa4ef4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:27.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:51:27.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:51:27.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:51:27.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:51:27.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:51:27.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:51:27.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:51:27.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:51:27.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:51:27.584 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:51:27.777 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:27.774+0000 7fefa4ef4ec0 -1 Falling back to public interface 2020-03-11T22:51:27.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:51:28.485 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:28.482+0000 7fefa4ef4ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:51:28.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:51:28.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:51:28.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:51:28.930 INFO:tasks.workunit.client.0.smithi145.stdout:1 2020-03-11T22:51:28.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:51:28.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:51:29.289 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/63719,v1:127.0.0.1:6819/63719] [v2:127.0.0.1:6820/63719,v1:127.0.0.1:6821/63719] exists,up 718bb608-6aa5-4b8d-8b2e-933157f1a615 2020-03-11T22:51:29.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:51:29.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:51:29.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:51:29.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:51:29.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 3 2020-03-11T22:51:29.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:51:29.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:51:29.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=3 2020-03-11T22:51:29.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:51:29.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/3 2020-03-11T22:51:29.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=9ff197a8-fd34-42fc-bffc-e43264b5b9b7 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:51:29.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:51:29.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:51:29.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:51:29.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/3' 2020-03-11T22:51:29.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/3/journal' 2020-03-11T22:51:29.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:51:29.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:51:29.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:51:29.291 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:51:29.291 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:51:29.291 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:51:29.292 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:51:29.292 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:29.292 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56096 2020-03-11T22:51:29.292 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56096/$cluster-$name.asok' 2020-03-11T22:51:29.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56096/$cluster-$name.asok' 2020-03-11T22:51:29.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:51:29.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:51:29.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:51:29.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:51:29.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:51:29.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:51:29.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:51:29.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:51:29.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:51:29.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:51:29.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/3 2020-03-11T22:51:29.294 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:51:29.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=3c1515e1-b675-4717-9d35-c641ec6e61b3 2020-03-11T22:51:29.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd3 3c1515e1-b675-4717-9d35-c641ec6e61b3' 2020-03-11T22:51:29.295 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:51:29.295 INFO:tasks.workunit.client.0.smithi145.stdout:add osd3 3c1515e1-b675-4717-9d35-c641ec6e61b3 2020-03-11T22:51:29.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBxa2leU4IJEhAAHfwuQ5Wk/p4VEXuo4uOAzA== 2020-03-11T22:51:29.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBxa2leU4IJEhAAHfwuQ5Wk/p4VEXuo4uOAzA=="}' 2020-03-11T22:51:29.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 3c1515e1-b675-4717-9d35-c641ec6e61b3 -i td/osd-backfill-prio/3/new.json 2020-03-11T22:51:29.790 INFO:tasks.workunit.client.0.smithi145.stdout:3 2020-03-11T22:51:29.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/3/new.json 2020-03-11T22:51:29.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 3 --fsid=9ff197a8-fd34-42fc-bffc-e43264b5b9b7 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/3 --osd-journal=td/osd-backfill-prio/3/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56096/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBxa2leU4IJEhAAHfwuQ5Wk/p4VEXuo4uOAzA== --osd-uuid 3c1515e1-b675-4717-9d35-c641ec6e61b3 2020-03-11T22:51:29.826 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:29.823+0000 7f288da2dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:29.828 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:29.825+0000 7f288da2dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:29.829 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:29.826+0000 7f288da2dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:29.830 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:29.827+0000 7f288da2dec0 -1 bluestore(td/osd-backfill-prio/3/block) _read_bdev_label failed to open td/osd-backfill-prio/3/block: (2) No such file or directory 2020-03-11T22:51:29.830 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:29.827+0000 7f288da2dec0 -1 bluestore(td/osd-backfill-prio/3/block) _read_bdev_label failed to open td/osd-backfill-prio/3/block: (2) No such file or directory 2020-03-11T22:51:29.830 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:29.827+0000 7f288da2dec0 -1 bluestore(td/osd-backfill-prio/3/block) _read_bdev_label failed to open td/osd-backfill-prio/3/block: (2) No such file or directory 2020-03-11T22:51:29.870 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:29.867+0000 7f288da2dec0 -1 bluestore(td/osd-backfill-prio/3) _read_fsid unparsable uuid 2020-03-11T22:51:32.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/3/keyring 2020-03-11T22:51:32.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:51:32.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd3 key to auth repository 2020-03-11T22:51:32.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:51:32.887 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd3 key to auth repository 2020-03-11T22:51:33.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.3 2020-03-11T22:51:33.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 3 --fsid=9ff197a8-fd34-42fc-bffc-e43264b5b9b7 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/3 --osd-journal=td/osd-backfill-prio/3/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56096/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:51:33.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:51:33.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:51:33.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:51:33.351 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.3 2020-03-11T22:51:33.374 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:33.371+0000 7f3dc1c9fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:33.375 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:33.373+0000 7f3dc1c9fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:33.377 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:33.374+0000 7f3dc1c9fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:33.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 3 2020-03-11T22:51:33.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:51:33.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=3 2020-03-11T22:51:33.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:51:33.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:51:33.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:51:33.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:51:33.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:51:33.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:51:33.696 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:51:33.897 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:33.894+0000 7f3dc1c9fec0 -1 Falling back to public interface 2020-03-11T22:51:34.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:51:34.676 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:34.674+0000 7f3dc1c9fec0 -1 osd.3 0 log_to_monitors {default=true} 2020-03-11T22:51:35.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:51:35.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:51:35.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:51:35.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:51:35.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:51:35.044 INFO:tasks.workunit.client.0.smithi145.stdout:1 2020-03-11T22:51:35.412 INFO:tasks.workunit.client.0.smithi145.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/64074,v1:127.0.0.1:6827/64074] [v2:127.0.0.1:6828/64074,v1:127.0.0.1:6829/64074] exists,up 3c1515e1-b675-4717-9d35-c641ec6e61b3 2020-03-11T22:51:35.412 INFO:tasks.workunit.client.0.smithi145.stdout:add osd4 428ee403-7176-4bb6-8a8c-23489262be6d 2020-03-11T22:51:35.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:51:35.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:51:35.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:51:35.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:51:35.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 4 2020-03-11T22:51:35.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:51:35.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:51:35.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=4 2020-03-11T22:51:35.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:51:35.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/4 2020-03-11T22:51:35.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=9ff197a8-fd34-42fc-bffc-e43264b5b9b7 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:51:35.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:51:35.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:51:35.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:51:35.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/4' 2020-03-11T22:51:35.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/4/journal' 2020-03-11T22:51:35.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:51:35.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:51:35.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:51:35.415 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:51:35.415 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:51:35.415 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:51:35.415 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:51:35.416 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:35.416 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56096 2020-03-11T22:51:35.416 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56096/$cluster-$name.asok' 2020-03-11T22:51:35.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56096/$cluster-$name.asok' 2020-03-11T22:51:35.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:51:35.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:51:35.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:51:35.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:51:35.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:51:35.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:51:35.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:51:35.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:51:35.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:51:35.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:51:35.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/4 2020-03-11T22:51:35.418 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:51:35.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=428ee403-7176-4bb6-8a8c-23489262be6d 2020-03-11T22:51:35.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd4 428ee403-7176-4bb6-8a8c-23489262be6d' 2020-03-11T22:51:35.418 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:51:35.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQB3a2leoUpfGRAA/5GDDAYmumr3RE7+YbfZow== 2020-03-11T22:51:35.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQB3a2leoUpfGRAA/5GDDAYmumr3RE7+YbfZow=="}' 2020-03-11T22:51:35.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 428ee403-7176-4bb6-8a8c-23489262be6d -i td/osd-backfill-prio/4/new.json 2020-03-11T22:51:36.015 INFO:tasks.workunit.client.0.smithi145.stdout:4 2020-03-11T22:51:36.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/4/new.json 2020-03-11T22:51:36.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 4 --fsid=9ff197a8-fd34-42fc-bffc-e43264b5b9b7 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/4 --osd-journal=td/osd-backfill-prio/4/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56096/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB3a2leoUpfGRAA/5GDDAYmumr3RE7+YbfZow== --osd-uuid 428ee403-7176-4bb6-8a8c-23489262be6d 2020-03-11T22:51:36.051 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:36.048+0000 7fe2d92efec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:36.052 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:36.049+0000 7fe2d92efec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:36.054 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:36.051+0000 7fe2d92efec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:36.054 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:36.051+0000 7fe2d92efec0 -1 bluestore(td/osd-backfill-prio/4/block) _read_bdev_label failed to open td/osd-backfill-prio/4/block: (2) No such file or directory 2020-03-11T22:51:36.054 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:36.051+0000 7fe2d92efec0 -1 bluestore(td/osd-backfill-prio/4/block) _read_bdev_label failed to open td/osd-backfill-prio/4/block: (2) No such file or directory 2020-03-11T22:51:36.055 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:36.051+0000 7fe2d92efec0 -1 bluestore(td/osd-backfill-prio/4/block) _read_bdev_label failed to open td/osd-backfill-prio/4/block: (2) No such file or directory 2020-03-11T22:51:36.101 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:36.098+0000 7fe2d92efec0 -1 bluestore(td/osd-backfill-prio/4) _read_fsid unparsable uuid 2020-03-11T22:51:39.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/4/keyring 2020-03-11T22:51:39.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:51:39.188 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd4 key to auth repository 2020-03-11T22:51:39.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd4 key to auth repository 2020-03-11T22:51:39.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:51:39.646 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.4 2020-03-11T22:51:39.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.4 2020-03-11T22:51:39.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 4 --fsid=9ff197a8-fd34-42fc-bffc-e43264b5b9b7 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/4 --osd-journal=td/osd-backfill-prio/4/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56096/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:51:39.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:51:39.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:51:39.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:51:39.669 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:39.666+0000 7f0100cf0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:39.671 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:39.668+0000 7f0100cf0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:39.673 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:39.670+0000 7f0100cf0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:40.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 4 2020-03-11T22:51:40.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:51:40.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=4 2020-03-11T22:51:40.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:51:40.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:51:40.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:51:40.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:51:40.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:51:40.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-03-11T22:51:40.003 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:51:40.188 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:40.185+0000 7f0100cf0ec0 -1 Falling back to public interface 2020-03-11T22:51:40.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:51:40.892 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:51:40.889+0000 7f0100cf0ec0 -1 osd.4 0 log_to_monitors {default=true} 2020-03-11T22:51:41.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:51:41.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:51:41.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:51:41.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:51:41.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-03-11T22:51:41.391 INFO:tasks.workunit.client.0.smithi145.stdout:1 2020-03-11T22:51:41.746 INFO:tasks.workunit.client.0.smithi145.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/64429,v1:127.0.0.1:6835/64429] [v2:127.0.0.1:6836/64429,v1:127.0.0.1:6837/64429] exists,up 428ee403-7176-4bb6-8a8c-23489262be6d 2020-03-11T22:51:41.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:51:41.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:51:41.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:51:41.747 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: seq 1 10 2020-03-11T22:51:41.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:51:41.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test1 1 1 2020-03-11T22:51:41.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test1 1 1 2020-03-11T22:51:42.324 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test1' already exists 2020-03-11T22:51:42.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:51:43.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test1 size 2 2020-03-11T22:51:44.032 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 2 2020-03-11T22:51:44.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:51:44.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test2 1 1 2020-03-11T22:51:44.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test2 1 1 2020-03-11T22:51:44.693 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test2' already exists 2020-03-11T22:51:44.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:51:45.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test2 size 2 2020-03-11T22:51:46.396 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 2 size to 2 2020-03-11T22:51:46.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:51:46.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test3 1 1 2020-03-11T22:51:46.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test3 1 1 2020-03-11T22:51:46.937 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test3' already exists 2020-03-11T22:51:46.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:51:47.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test3 size 2 2020-03-11T22:51:48.535 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 3 size to 2 2020-03-11T22:51:48.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:51:48.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test4 1 1 2020-03-11T22:51:48.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test4 1 1 2020-03-11T22:51:49.180 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test4' already exists 2020-03-11T22:51:49.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:51:50.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test4 size 2 2020-03-11T22:51:50.934 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 4 size to 2 2020-03-11T22:51:50.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:51:50.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test5 1 1 2020-03-11T22:51:50.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test5 1 1 2020-03-11T22:51:51.443 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test5' already exists 2020-03-11T22:51:51.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:51:52.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test5 size 2 2020-03-11T22:51:53.081 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 5 size to 2 2020-03-11T22:51:53.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:51:53.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test6 1 1 2020-03-11T22:51:53.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test6 1 1 2020-03-11T22:51:53.665 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test6' already exists 2020-03-11T22:51:53.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:51:54.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test6 size 2 2020-03-11T22:51:55.575 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 6 size to 2 2020-03-11T22:51:55.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:51:55.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test7 1 1 2020-03-11T22:51:55.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test7 1 1 2020-03-11T22:51:56.087 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test7' already exists 2020-03-11T22:51:56.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:51:57.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test7 size 2 2020-03-11T22:51:58.037 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 7 size to 2 2020-03-11T22:51:58.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:51:58.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test8 1 1 2020-03-11T22:51:58.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test8 1 1 2020-03-11T22:51:58.621 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test8' already exists 2020-03-11T22:51:58.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:51:59.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test8 size 2 2020-03-11T22:52:00.468 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 8 size to 2 2020-03-11T22:52:00.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:00.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test9 1 1 2020-03-11T22:52:00.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test9 1 1 2020-03-11T22:52:00.966 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test9' already exists 2020-03-11T22:52:00.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:52:01.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test9 size 2 2020-03-11T22:52:02.698 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 9 size to 2 2020-03-11T22:52:02.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:02.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test10 1 1 2020-03-11T22:52:02.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test10 1 1 2020-03-11T22:52:03.189 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test10' already exists 2020-03-11T22:52:03.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:52:04.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test10 size 2 2020-03-11T22:52:04.919 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 10 size to 2 2020-03-11T22:52:04.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:67: TEST_backfill_priority: sleep 5 2020-03-11T22:52:09.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:69: TEST_backfill_priority: wait_for_clean 2020-03-11T22:52:09.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:52:09.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:52:09.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:52:09.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:52:09.930 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:52:09.930 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:52:09.930 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:52:09.931 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:52:09.931 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:52:09.931 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:52:10.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:52:10.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:52:10.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:52:10.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:52:10.033 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:52:10.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:52:10.379 INFO:tasks.workunit.client.0.smithi145.stderr:1 2020-03-11T22:52:10.380 INFO:tasks.workunit.client.0.smithi145.stderr:2 2020-03-11T22:52:10.380 INFO:tasks.workunit.client.0.smithi145.stderr:3 2020-03-11T22:52:10.380 INFO:tasks.workunit.client.0.smithi145.stderr:4' 2020-03-11T22:52:10.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:52:10.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:52:10.380 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:52:10.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836493 2020-03-11T22:52:10.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493' 2020-03-11T22:52:10.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:52:10.506 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:52:10.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672973 2020-03-11T22:52:10.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672973' 2020-03-11T22:52:10.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:52:10.632 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:52:10.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509451 2020-03-11T22:52:10.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672973 2-64424509451' 2020-03-11T22:52:10.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:52:10.759 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:52:10.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345929 2020-03-11T22:52:10.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672973 2-64424509451 3-85899345929' 2020-03-11T22:52:10.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:52:10.885 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:52:11.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182408 2020-03-11T22:52:11.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672973 2-64424509451 3-85899345929 4-107374182408' 2020-03-11T22:52:11.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:52:11.013 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836493 2020-03-11T22:52:11.013 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:52:11.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:52:11.016 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836493 2020-03-11T22:52:11.016 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:52:11.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836493 2020-03-11T22:52:11.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2020-03-11T22:52:11.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:52:11.017 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836493 2020-03-11T22:52:11.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836491 -lt 21474836493 2020-03-11T22:52:11.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:52:12.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:52:12.360 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:52:12.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836493 -lt 21474836493 2020-03-11T22:52:12.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:52:12.713 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672973 2020-03-11T22:52:12.713 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:52:12.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:52:12.714 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:52:12.714 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672973 2020-03-11T22:52:12.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672973 2020-03-11T22:52:12.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2020-03-11T22:52:12.715 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:52:12.715 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672973 2020-03-11T22:52:13.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672973 -lt 42949672973 2020-03-11T22:52:13.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:52:13.062 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509451 2020-03-11T22:52:13.062 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:52:13.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:52:13.063 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509451 2020-03-11T22:52:13.063 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:52:13.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509451 2020-03-11T22:52:13.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2020-03-11T22:52:13.065 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509451 2020-03-11T22:52:13.065 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:52:13.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509451 -lt 64424509451 2020-03-11T22:52:13.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:52:13.476 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345929 2020-03-11T22:52:13.476 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:52:13.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:52:13.477 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345929 2020-03-11T22:52:13.478 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:52:13.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345929 2020-03-11T22:52:13.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345929' 2020-03-11T22:52:13.479 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:52:13.479 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345929 2020-03-11T22:52:13.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345930 -lt 85899345929 2020-03-11T22:52:13.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:52:13.828 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182408 2020-03-11T22:52:13.829 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:52:13.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:52:13.829 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182408 2020-03-11T22:52:13.830 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:52:13.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182408 2020-03-11T22:52:13.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182408' 2020-03-11T22:52:13.831 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:52:13.831 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182408 2020-03-11T22:52:14.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182408 -lt 107374182408 2020-03-11T22:52:14.175 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:52:14.176 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:14.176 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:14.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 10 == 0 2020-03-11T22:52:14.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:14.624 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:14.624 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:14.624 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:14.624 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:14.625 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:14.625 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:14.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=10 2020-03-11T22:52:14.955 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:14.955 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:14.955 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:15.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 10 = 10 2020-03-11T22:52:15.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:52:15.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:52:15.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:71: TEST_backfill_priority: ceph pg dump pgs 2020-03-11T22:52:15.721 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:52:15.721 INFO:tasks.workunit.client.0.smithi145.stdout:10.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:52:05.263782+0000 0'0 83:20 [2,1] 2 [2,1] 2 0'0 2020-03-11T22:52:03.010780+0000 0'0 2020-03-11T22:52:03.010780+0000 0 2020-03-11T22:52:15.721 INFO:tasks.workunit.client.0.smithi145.stdout:9.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:52:03.548647+0000 0'0 81:23 [1,2] 1 [1,2] 1 0'0 2020-03-11T22:52:00.806343+0000 0'0 2020-03-11T22:52:00.806343+0000 0 2020-03-11T22:52:15.722 INFO:tasks.workunit.client.0.smithi145.stdout:8.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:52:00.740489+0000 0'0 80:26 [3,4] 3 [3,4] 3 0'0 2020-03-11T22:51:58.399275+0000 0'0 2020-03-11T22:51:58.399275+0000 0 2020-03-11T22:52:15.722 INFO:tasks.workunit.client.0.smithi145.stdout:7.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:51:58.107483+0000 0'0 81:34 [1,4] 1 [1,4] 1 0'0 2020-03-11T22:51:55.888484+0000 0'0 2020-03-11T22:51:55.888484+0000 0 2020-03-11T22:52:15.722 INFO:tasks.workunit.client.0.smithi145.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:51:46.987674+0000 0'0 80:51 [3,1] 3 [3,1] 3 0'0 2020-03-11T22:51:44.420056+0000 0'0 2020-03-11T22:51:44.420056+0000 0 2020-03-11T22:52:15.722 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:51:44.073807+0000 0'0 81:50 [1,0] 1 [1,0] 1 0'0 2020-03-11T22:51:42.130628+0000 0'0 2020-03-11T22:51:42.130628+0000 0 2020-03-11T22:52:15.722 INFO:tasks.workunit.client.0.smithi145.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:51:49.057950+0000 0'0 81:48 [1,2] 1 [1,2] 1 0'0 2020-03-11T22:51:46.721759+0000 0'0 2020-03-11T22:51:46.721759+0000 0 2020-03-11T22:52:15.722 INFO:tasks.workunit.client.0.smithi145.stdout:4.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:51:51.283700+0000 0'0 82:45 [4,0] 4 [4,0] 4 0'0 2020-03-11T22:51:48.839071+0000 0'0 2020-03-11T22:51:48.839071+0000 0 2020-03-11T22:52:15.723 INFO:tasks.workunit.client.0.smithi145.stdout:5.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:51:53.264778+0000 0'0 80:33 [3,4] 3 [3,4] 3 0'0 2020-03-11T22:51:51.247490+0000 0'0 2020-03-11T22:51:51.247490+0000 0 2020-03-11T22:52:15.723 INFO:tasks.workunit.client.0.smithi145.stdout:6.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:51:55.689974+0000 0'0 82:36 [4,2] 4 [4,2] 4 0'0 2020-03-11T22:51:53.485711+0000 0'0 2020-03-11T22:51:53.485711+0000 0 2020-03-11T22:52:15.723 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:52:15.723 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:52:15.724 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:52:15.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:75: TEST_backfill_priority: local PG1 2020-03-11T22:52:15.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:76: TEST_backfill_priority: local POOLNUM1 2020-03-11T22:52:15.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:77: TEST_backfill_priority: local pool1 2020-03-11T22:52:15.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:78: TEST_backfill_priority: local chk_osd1_1 2020-03-11T22:52:15.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:79: TEST_backfill_priority: local chk_osd1_2 2020-03-11T22:52:15.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:81: TEST_backfill_priority: local PG2 2020-03-11T22:52:15.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:82: TEST_backfill_priority: local POOLNUM2 2020-03-11T22:52:15.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:83: TEST_backfill_priority: local pool2 2020-03-11T22:52:15.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:84: TEST_backfill_priority: local chk_osd2 2020-03-11T22:52:15.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:86: TEST_backfill_priority: local PG3 2020-03-11T22:52:15.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:87: TEST_backfill_priority: local POOLNUM3 2020-03-11T22:52:15.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:88: TEST_backfill_priority: local pool3 2020-03-11T22:52:15.733 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: seq 1 10 2020-03-11T22:52:15.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:15.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 1.0 --format=json 2020-03-11T22:52:15.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:52:16.070 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:52:16.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=1 2020-03-11T22:52:16.071 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:52:16.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=0 2020-03-11T22:52:16.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z '' ']' 2020-03-11T22:52:16.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:97: TEST_backfill_priority: PG1=1.0 2020-03-11T22:52:16.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:98: TEST_backfill_priority: POOLNUM1=1 2020-03-11T22:52:16.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:99: TEST_backfill_priority: pool1=test1 2020-03-11T22:52:16.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:100: TEST_backfill_priority: chk_osd1_1=1 2020-03-11T22:52:16.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:101: TEST_backfill_priority: chk_osd1_2=0 2020-03-11T22:52:16.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:16.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 2.0 --format=json 2020-03-11T22:52:16.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:52:16.407 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:52:16.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=3 2020-03-11T22:52:16.408 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:52:16.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=1 2020-03-11T22:52:16.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:52:16.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z '' -a 1 = 3 -a 0 '!=' 1 ']' 2020-03-11T22:52:16.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n '' -a 1 = 3 -a 0 '!=' 1 -a '' '!=' 1 ']' 2020-03-11T22:52:16.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:16.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 3.0 --format=json 2020-03-11T22:52:16.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:52:16.749 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:52:16.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=1 2020-03-11T22:52:16.750 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:52:16.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=2 2020-03-11T22:52:16.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:52:16.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z '' -a 1 = 1 -a 0 '!=' 2 ']' 2020-03-11T22:52:16.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:104: TEST_backfill_priority: PG2=3.0 2020-03-11T22:52:16.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:105: TEST_backfill_priority: POOLNUM2=3 2020-03-11T22:52:16.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:106: TEST_backfill_priority: pool2=test3 2020-03-11T22:52:16.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:107: TEST_backfill_priority: chk_osd2=2 2020-03-11T22:52:16.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:16.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 4.0 --format=json 2020-03-11T22:52:16.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:52:17.096 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:52:17.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=4 2020-03-11T22:52:17.097 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:52:17.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=0 2020-03-11T22:52:17.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:52:17.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 0 ']' 2020-03-11T22:52:17.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 0 -a 2 '!=' 0 ']' 2020-03-11T22:52:17.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:17.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 5.0 --format=json 2020-03-11T22:52:17.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:52:17.435 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:52:17.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=3 2020-03-11T22:52:17.436 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:52:17.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=4 2020-03-11T22:52:17.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:52:17.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 3 -a 0 '!=' 4 ']' 2020-03-11T22:52:17.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 3 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2020-03-11T22:52:17.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:17.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 6.0 --format=json 2020-03-11T22:52:17.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:52:17.773 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:52:17.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=4 2020-03-11T22:52:17.774 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:52:17.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=2 2020-03-11T22:52:17.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:52:17.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 2 ']' 2020-03-11T22:52:17.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 2 -a 2 '!=' 2 ']' 2020-03-11T22:52:17.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:17.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 7.0 --format=json 2020-03-11T22:52:17.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:52:18.111 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:52:18.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=1 2020-03-11T22:52:18.112 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:52:18.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=4 2020-03-11T22:52:18.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:52:18.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 1 -a 0 '!=' 4 ']' 2020-03-11T22:52:18.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 1 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2020-03-11T22:52:18.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:110: TEST_backfill_priority: PG3=7.0 2020-03-11T22:52:18.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:111: TEST_backfill_priority: POOLNUM3=7 2020-03-11T22:52:18.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:112: TEST_backfill_priority: pool3=test7 2020-03-11T22:52:18.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:113: TEST_backfill_priority: break 2020-03-11T22:52:18.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:116: TEST_backfill_priority: rm -f td/osd-backfill-prio/acting 2020-03-11T22:52:18.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:118: TEST_backfill_priority: '[' test3 = '' -o pool3 = '' ']' 2020-03-11T22:52:18.116 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: seq 1 10 2020-03-11T22:52:18.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:18.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 1 '!=' 1 -a 1 '!=' 3 -a 1 '!=' 7 ']' 2020-03-11T22:52:18.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:18.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 2 '!=' 1 -a 2 '!=' 3 -a 2 '!=' 7 ']' 2020-03-11T22:52:18.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test2 2020-03-11T22:52:18.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test2 2020-03-11T22:52:18.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2020-03-11T22:52:18.648 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test2' does not exist 2020-03-11T22:52:18.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:18.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 3 '!=' 1 -a 3 '!=' 3 -a 3 '!=' 7 ']' 2020-03-11T22:52:18.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:18.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 4 '!=' 1 -a 4 '!=' 3 -a 4 '!=' 7 ']' 2020-03-11T22:52:18.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test4 2020-03-11T22:52:18.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test4 2020-03-11T22:52:18.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test4 test4 --yes-i-really-really-mean-it 2020-03-11T22:52:19.216 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test4' does not exist 2020-03-11T22:52:19.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:19.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 5 '!=' 1 -a 5 '!=' 3 -a 5 '!=' 7 ']' 2020-03-11T22:52:19.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test5 2020-03-11T22:52:19.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test5 2020-03-11T22:52:19.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test5 test5 --yes-i-really-really-mean-it 2020-03-11T22:52:19.814 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test5' does not exist 2020-03-11T22:52:19.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:19.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 6 '!=' 1 -a 6 '!=' 3 -a 6 '!=' 7 ']' 2020-03-11T22:52:19.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test6 2020-03-11T22:52:19.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test6 2020-03-11T22:52:19.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test6 test6 --yes-i-really-really-mean-it 2020-03-11T22:52:20.320 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test6' does not exist 2020-03-11T22:52:20.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:20.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 7 '!=' 1 -a 7 '!=' 3 -a 7 '!=' 7 ']' 2020-03-11T22:52:20.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:20.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 8 '!=' 1 -a 8 '!=' 3 -a 8 '!=' 7 ']' 2020-03-11T22:52:20.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test8 2020-03-11T22:52:20.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test8 2020-03-11T22:52:20.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test8 test8 --yes-i-really-really-mean-it 2020-03-11T22:52:20.829 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test8' does not exist 2020-03-11T22:52:20.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:20.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 9 '!=' 1 -a 9 '!=' 3 -a 9 '!=' 7 ']' 2020-03-11T22:52:20.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test9 2020-03-11T22:52:20.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test9 2020-03-11T22:52:20.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test9 test9 --yes-i-really-really-mean-it 2020-03-11T22:52:21.347 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test9' does not exist 2020-03-11T22:52:21.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:21.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 10 '!=' 1 -a 10 '!=' 3 -a 10 '!=' 7 ']' 2020-03-11T22:52:21.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test10 2020-03-11T22:52:21.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test10 2020-03-11T22:52:21.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test10 test10 --yes-i-really-really-mean-it 2020-03-11T22:52:21.940 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test10' does not exist 2020-03-11T22:52:21.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:132: TEST_backfill_priority: ceph osd pool set test3 size 1 2020-03-11T22:52:22.668 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 3 size to 1 2020-03-11T22:52:22.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:133: TEST_backfill_priority: ceph osd pool set test7 size 1 2020-03-11T22:52:23.497 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 7 size to 1 2020-03-11T22:52:23.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:134: TEST_backfill_priority: wait_for_clean 2020-03-11T22:52:23.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:52:23.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:52:23.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:52:23.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:52:23.506 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:52:23.507 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:52:23.507 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:52:23.508 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:52:23.508 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:52:23.508 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:52:23.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:52:23.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:52:23.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:52:23.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:52:23.597 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:52:23.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:52:23.946 INFO:tasks.workunit.client.0.smithi145.stderr:1 2020-03-11T22:52:23.946 INFO:tasks.workunit.client.0.smithi145.stderr:2 2020-03-11T22:52:23.947 INFO:tasks.workunit.client.0.smithi145.stderr:3 2020-03-11T22:52:23.947 INFO:tasks.workunit.client.0.smithi145.stderr:4' 2020-03-11T22:52:23.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:52:23.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:52:23.947 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:52:24.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836498 2020-03-11T22:52:24.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836498' 2020-03-11T22:52:24.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:52:24.074 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:52:24.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672978 2020-03-11T22:52:24.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836498 1-42949672978' 2020-03-11T22:52:24.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:52:24.200 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:52:24.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509455 2020-03-11T22:52:24.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836498 1-42949672978 2-64424509455' 2020-03-11T22:52:24.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:52:24.326 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:52:24.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345934 2020-03-11T22:52:24.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836498 1-42949672978 2-64424509455 3-85899345934' 2020-03-11T22:52:24.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:52:24.451 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:52:24.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182413 2020-03-11T22:52:24.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836498 1-42949672978 2-64424509455 3-85899345934 4-107374182413' 2020-03-11T22:52:24.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:52:24.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836498 2020-03-11T22:52:24.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:52:24.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:52:24.578 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836498 2020-03-11T22:52:24.578 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:52:24.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836498 2020-03-11T22:52:24.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2020-03-11T22:52:24.579 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:52:24.580 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836498 2020-03-11T22:52:24.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836498 -lt 21474836498 2020-03-11T22:52:24.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:52:24.925 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672978 2020-03-11T22:52:24.925 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:52:24.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:52:24.926 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672978 2020-03-11T22:52:24.926 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:52:24.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672978 2020-03-11T22:52:24.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2020-03-11T22:52:24.927 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:52:24.928 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672978 2020-03-11T22:52:25.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672976 -lt 42949672978 2020-03-11T22:52:25.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:52:26.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:52:26.299 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:52:26.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672978 -lt 42949672978 2020-03-11T22:52:26.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:52:26.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509455 2020-03-11T22:52:26.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:52:26.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:52:26.654 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509455 2020-03-11T22:52:26.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:52:26.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509455 2020-03-11T22:52:26.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2020-03-11T22:52:26.656 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:52:26.656 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509455 2020-03-11T22:52:27.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509456 -lt 64424509455 2020-03-11T22:52:27.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:52:27.004 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345934 2020-03-11T22:52:27.004 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:52:27.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:52:27.005 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345934 2020-03-11T22:52:27.005 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:52:27.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345934 2020-03-11T22:52:27.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345934' 2020-03-11T22:52:27.007 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:52:27.007 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345934 2020-03-11T22:52:27.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345935 -lt 85899345934 2020-03-11T22:52:27.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:52:27.366 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182413 2020-03-11T22:52:27.366 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:52:27.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:52:27.367 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182413 2020-03-11T22:52:27.367 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:52:27.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182413 2020-03-11T22:52:27.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182413' 2020-03-11T22:52:27.368 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:52:27.368 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182413 2020-03-11T22:52:27.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182413 -lt 107374182413 2020-03-11T22:52:27.714 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:52:27.714 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:27.715 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:28.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 3 == 0 2020-03-11T22:52:28.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:28.159 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:28.159 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:28.159 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:28.159 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:28.160 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:28.160 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:28.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=3 2020-03-11T22:52:28.496 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:28.497 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:28.497 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:28.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 3 = 3 2020-03-11T22:52:28.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:52:28.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:52:28.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:136: TEST_backfill_priority: dd if=/dev/urandom of=td/osd-backfill-prio/data bs=1M count=10 2020-03-11T22:52:28.988 INFO:tasks.workunit.client.0.smithi145.stderr:10+0 records in 2020-03-11T22:52:28.988 INFO:tasks.workunit.client.0.smithi145.stderr:10+0 records out 2020-03-11T22:52:28.988 INFO:tasks.workunit.client.0.smithi145.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0455972 s, 230 MB/s 2020-03-11T22:52:28.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:137: TEST_backfill_priority: p=1 2020-03-11T22:52:28.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:138: TEST_backfill_priority: for pname in $pool1 $pool2 $pool3 2020-03-11T22:52:28.989 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: seq 1 50 2020-03-11T22:52:28.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:28.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj1-p1 td/osd-backfill-prio/data 2020-03-11T22:52:29.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:29.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj2-p1 td/osd-backfill-prio/data 2020-03-11T22:52:29.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:29.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj3-p1 td/osd-backfill-prio/data 2020-03-11T22:52:30.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:30.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj4-p1 td/osd-backfill-prio/data 2020-03-11T22:52:30.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:30.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj5-p1 td/osd-backfill-prio/data 2020-03-11T22:52:31.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:31.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj6-p1 td/osd-backfill-prio/data 2020-03-11T22:52:31.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:31.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj7-p1 td/osd-backfill-prio/data 2020-03-11T22:52:31.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:31.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj8-p1 td/osd-backfill-prio/data 2020-03-11T22:52:32.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:32.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj9-p1 td/osd-backfill-prio/data 2020-03-11T22:52:32.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:32.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj10-p1 td/osd-backfill-prio/data 2020-03-11T22:52:33.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:33.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj11-p1 td/osd-backfill-prio/data 2020-03-11T22:52:33.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:33.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj12-p1 td/osd-backfill-prio/data 2020-03-11T22:52:34.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:34.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj13-p1 td/osd-backfill-prio/data 2020-03-11T22:52:34.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:34.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj14-p1 td/osd-backfill-prio/data 2020-03-11T22:52:35.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:35.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj15-p1 td/osd-backfill-prio/data 2020-03-11T22:52:35.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:35.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj16-p1 td/osd-backfill-prio/data 2020-03-11T22:52:36.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:36.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj17-p1 td/osd-backfill-prio/data 2020-03-11T22:52:36.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:36.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj18-p1 td/osd-backfill-prio/data 2020-03-11T22:52:36.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:36.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj19-p1 td/osd-backfill-prio/data 2020-03-11T22:52:37.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:37.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj20-p1 td/osd-backfill-prio/data 2020-03-11T22:52:37.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:37.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj21-p1 td/osd-backfill-prio/data 2020-03-11T22:52:38.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:38.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj22-p1 td/osd-backfill-prio/data 2020-03-11T22:52:38.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:38.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj23-p1 td/osd-backfill-prio/data 2020-03-11T22:52:39.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:39.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj24-p1 td/osd-backfill-prio/data 2020-03-11T22:52:39.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:39.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj25-p1 td/osd-backfill-prio/data 2020-03-11T22:52:40.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:40.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj26-p1 td/osd-backfill-prio/data 2020-03-11T22:52:40.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:40.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj27-p1 td/osd-backfill-prio/data 2020-03-11T22:52:40.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:40.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj28-p1 td/osd-backfill-prio/data 2020-03-11T22:52:41.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:41.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj29-p1 td/osd-backfill-prio/data 2020-03-11T22:52:41.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:41.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj30-p1 td/osd-backfill-prio/data 2020-03-11T22:52:42.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:42.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj31-p1 td/osd-backfill-prio/data 2020-03-11T22:52:42.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:42.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj32-p1 td/osd-backfill-prio/data 2020-03-11T22:52:43.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:43.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj33-p1 td/osd-backfill-prio/data 2020-03-11T22:52:43.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:43.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj34-p1 td/osd-backfill-prio/data 2020-03-11T22:52:44.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:44.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj35-p1 td/osd-backfill-prio/data 2020-03-11T22:52:44.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:44.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj36-p1 td/osd-backfill-prio/data 2020-03-11T22:52:44.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:44.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj37-p1 td/osd-backfill-prio/data 2020-03-11T22:52:45.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:45.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj38-p1 td/osd-backfill-prio/data 2020-03-11T22:52:45.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:45.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj39-p1 td/osd-backfill-prio/data 2020-03-11T22:52:46.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:46.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj40-p1 td/osd-backfill-prio/data 2020-03-11T22:52:46.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:46.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj41-p1 td/osd-backfill-prio/data 2020-03-11T22:52:47.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:47.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj42-p1 td/osd-backfill-prio/data 2020-03-11T22:52:47.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:47.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj43-p1 td/osd-backfill-prio/data 2020-03-11T22:52:48.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:48.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj44-p1 td/osd-backfill-prio/data 2020-03-11T22:52:48.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:48.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj45-p1 td/osd-backfill-prio/data 2020-03-11T22:52:49.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:49.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj46-p1 td/osd-backfill-prio/data 2020-03-11T22:52:49.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:49.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj47-p1 td/osd-backfill-prio/data 2020-03-11T22:52:49.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:49.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj48-p1 td/osd-backfill-prio/data 2020-03-11T22:52:50.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:50.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj49-p1 td/osd-backfill-prio/data 2020-03-11T22:52:50.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:50.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj50-p1 td/osd-backfill-prio/data 2020-03-11T22:52:51.223 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: expr 1 + 1 2020-03-11T22:52:51.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: p=2 2020-03-11T22:52:51.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:138: TEST_backfill_priority: for pname in $pool1 $pool2 $pool3 2020-03-11T22:52:51.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: seq 1 50 2020-03-11T22:52:51.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:51.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj1-p2 td/osd-backfill-prio/data 2020-03-11T22:52:51.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:51.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj2-p2 td/osd-backfill-prio/data 2020-03-11T22:52:51.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:51.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj3-p2 td/osd-backfill-prio/data 2020-03-11T22:52:51.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:51.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj4-p2 td/osd-backfill-prio/data 2020-03-11T22:52:52.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:52.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj5-p2 td/osd-backfill-prio/data 2020-03-11T22:52:52.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:52.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj6-p2 td/osd-backfill-prio/data 2020-03-11T22:52:52.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:52.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj7-p2 td/osd-backfill-prio/data 2020-03-11T22:52:53.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:53.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj8-p2 td/osd-backfill-prio/data 2020-03-11T22:52:53.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:53.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj9-p2 td/osd-backfill-prio/data 2020-03-11T22:52:53.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:53.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj10-p2 td/osd-backfill-prio/data 2020-03-11T22:52:53.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:53.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj11-p2 td/osd-backfill-prio/data 2020-03-11T22:52:53.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:53.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj12-p2 td/osd-backfill-prio/data 2020-03-11T22:52:54.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:54.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj13-p2 td/osd-backfill-prio/data 2020-03-11T22:52:54.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:54.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj14-p2 td/osd-backfill-prio/data 2020-03-11T22:52:54.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:54.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj15-p2 td/osd-backfill-prio/data 2020-03-11T22:52:55.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:55.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj16-p2 td/osd-backfill-prio/data 2020-03-11T22:52:55.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:55.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj17-p2 td/osd-backfill-prio/data 2020-03-11T22:52:55.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:55.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj18-p2 td/osd-backfill-prio/data 2020-03-11T22:52:55.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:55.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj19-p2 td/osd-backfill-prio/data 2020-03-11T22:52:56.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:56.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj20-p2 td/osd-backfill-prio/data 2020-03-11T22:52:56.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:56.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj21-p2 td/osd-backfill-prio/data 2020-03-11T22:52:56.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:56.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj22-p2 td/osd-backfill-prio/data 2020-03-11T22:52:56.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:56.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj23-p2 td/osd-backfill-prio/data 2020-03-11T22:52:57.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:57.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj24-p2 td/osd-backfill-prio/data 2020-03-11T22:52:57.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:57.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj25-p2 td/osd-backfill-prio/data 2020-03-11T22:52:57.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:57.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj26-p2 td/osd-backfill-prio/data 2020-03-11T22:52:57.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:57.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj27-p2 td/osd-backfill-prio/data 2020-03-11T22:52:58.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:58.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj28-p2 td/osd-backfill-prio/data 2020-03-11T22:52:58.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:58.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj29-p2 td/osd-backfill-prio/data 2020-03-11T22:52:58.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:58.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj30-p2 td/osd-backfill-prio/data 2020-03-11T22:52:58.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:58.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj31-p2 td/osd-backfill-prio/data 2020-03-11T22:52:59.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:59.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj32-p2 td/osd-backfill-prio/data 2020-03-11T22:52:59.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:59.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj33-p2 td/osd-backfill-prio/data 2020-03-11T22:52:59.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:59.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj34-p2 td/osd-backfill-prio/data 2020-03-11T22:52:59.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:52:59.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj35-p2 td/osd-backfill-prio/data 2020-03-11T22:53:00.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:00.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj36-p2 td/osd-backfill-prio/data 2020-03-11T22:53:00.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:00.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj37-p2 td/osd-backfill-prio/data 2020-03-11T22:53:00.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:00.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj38-p2 td/osd-backfill-prio/data 2020-03-11T22:53:00.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:00.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj39-p2 td/osd-backfill-prio/data 2020-03-11T22:53:01.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:01.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj40-p2 td/osd-backfill-prio/data 2020-03-11T22:53:01.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:01.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj41-p2 td/osd-backfill-prio/data 2020-03-11T22:53:01.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:01.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj42-p2 td/osd-backfill-prio/data 2020-03-11T22:53:01.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:01.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj43-p2 td/osd-backfill-prio/data 2020-03-11T22:53:02.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:02.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj44-p2 td/osd-backfill-prio/data 2020-03-11T22:53:02.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:02.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj45-p2 td/osd-backfill-prio/data 2020-03-11T22:53:02.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:02.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj46-p2 td/osd-backfill-prio/data 2020-03-11T22:53:02.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:02.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj47-p2 td/osd-backfill-prio/data 2020-03-11T22:53:03.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:03.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj48-p2 td/osd-backfill-prio/data 2020-03-11T22:53:03.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:03.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj49-p2 td/osd-backfill-prio/data 2020-03-11T22:53:03.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:03.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj50-p2 td/osd-backfill-prio/data 2020-03-11T22:53:03.864 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: expr 2 + 1 2020-03-11T22:53:03.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: p=3 2020-03-11T22:53:03.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:138: TEST_backfill_priority: for pname in $pool1 $pool2 $pool3 2020-03-11T22:53:03.866 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: seq 1 50 2020-03-11T22:53:03.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:03.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj1-p3 td/osd-backfill-prio/data 2020-03-11T22:53:04.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:04.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj2-p3 td/osd-backfill-prio/data 2020-03-11T22:53:04.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:04.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj3-p3 td/osd-backfill-prio/data 2020-03-11T22:53:04.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:04.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj4-p3 td/osd-backfill-prio/data 2020-03-11T22:53:04.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:04.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj5-p3 td/osd-backfill-prio/data 2020-03-11T22:53:05.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:05.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj6-p3 td/osd-backfill-prio/data 2020-03-11T22:53:05.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:05.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj7-p3 td/osd-backfill-prio/data 2020-03-11T22:53:05.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:05.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj8-p3 td/osd-backfill-prio/data 2020-03-11T22:53:05.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:05.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj9-p3 td/osd-backfill-prio/data 2020-03-11T22:53:06.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:06.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj10-p3 td/osd-backfill-prio/data 2020-03-11T22:53:06.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:06.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj11-p3 td/osd-backfill-prio/data 2020-03-11T22:53:06.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:06.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj12-p3 td/osd-backfill-prio/data 2020-03-11T22:53:06.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:06.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj13-p3 td/osd-backfill-prio/data 2020-03-11T22:53:07.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:07.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj14-p3 td/osd-backfill-prio/data 2020-03-11T22:53:07.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:07.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj15-p3 td/osd-backfill-prio/data 2020-03-11T22:53:07.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:07.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj16-p3 td/osd-backfill-prio/data 2020-03-11T22:53:08.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:08.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj17-p3 td/osd-backfill-prio/data 2020-03-11T22:53:08.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:08.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj18-p3 td/osd-backfill-prio/data 2020-03-11T22:53:08.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:08.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj19-p3 td/osd-backfill-prio/data 2020-03-11T22:53:08.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:08.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj20-p3 td/osd-backfill-prio/data 2020-03-11T22:53:08.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:08.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj21-p3 td/osd-backfill-prio/data 2020-03-11T22:53:09.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:09.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj22-p3 td/osd-backfill-prio/data 2020-03-11T22:53:09.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:09.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj23-p3 td/osd-backfill-prio/data 2020-03-11T22:53:09.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:09.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj24-p3 td/osd-backfill-prio/data 2020-03-11T22:53:09.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:09.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj25-p3 td/osd-backfill-prio/data 2020-03-11T22:53:10.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:10.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj26-p3 td/osd-backfill-prio/data 2020-03-11T22:53:10.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:10.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj27-p3 td/osd-backfill-prio/data 2020-03-11T22:53:10.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:10.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj28-p3 td/osd-backfill-prio/data 2020-03-11T22:53:11.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:11.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj29-p3 td/osd-backfill-prio/data 2020-03-11T22:53:11.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:11.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj30-p3 td/osd-backfill-prio/data 2020-03-11T22:53:11.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:11.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj31-p3 td/osd-backfill-prio/data 2020-03-11T22:53:11.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:11.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj32-p3 td/osd-backfill-prio/data 2020-03-11T22:53:12.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:12.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj33-p3 td/osd-backfill-prio/data 2020-03-11T22:53:12.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:12.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj34-p3 td/osd-backfill-prio/data 2020-03-11T22:53:12.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:12.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj35-p3 td/osd-backfill-prio/data 2020-03-11T22:53:12.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:12.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj36-p3 td/osd-backfill-prio/data 2020-03-11T22:53:13.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:13.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj37-p3 td/osd-backfill-prio/data 2020-03-11T22:53:13.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:13.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj38-p3 td/osd-backfill-prio/data 2020-03-11T22:53:13.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:13.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj39-p3 td/osd-backfill-prio/data 2020-03-11T22:53:13.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:13.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj40-p3 td/osd-backfill-prio/data 2020-03-11T22:53:14.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:14.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj41-p3 td/osd-backfill-prio/data 2020-03-11T22:53:14.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:14.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj42-p3 td/osd-backfill-prio/data 2020-03-11T22:53:14.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:14.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj43-p3 td/osd-backfill-prio/data 2020-03-11T22:53:15.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:15.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj44-p3 td/osd-backfill-prio/data 2020-03-11T22:53:15.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:15.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj45-p3 td/osd-backfill-prio/data 2020-03-11T22:53:15.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:15.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj46-p3 td/osd-backfill-prio/data 2020-03-11T22:53:15.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:15.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj47-p3 td/osd-backfill-prio/data 2020-03-11T22:53:16.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:16.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj48-p3 td/osd-backfill-prio/data 2020-03-11T22:53:16.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:16.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj49-p3 td/osd-backfill-prio/data 2020-03-11T22:53:16.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:16.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj50-p3 td/osd-backfill-prio/data 2020-03-11T22:53:16.794 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: expr 3 + 1 2020-03-11T22:53:16.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: p=4 2020-03-11T22:53:16.796 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:147: TEST_backfill_priority: get_not_primary test1 obj1-p1 2020-03-11T22:53:16.796 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1166: get_not_primary: local poolname=test1 2020-03-11T22:53:16.796 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local objectname=obj1-p1 2020-03-11T22:53:16.796 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: get_primary test1 obj1-p1 2020-03-11T22:53:16.797 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: get_primary: local poolname=test1 2020-03-11T22:53:16.797 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local objectname=obj1-p1 2020-03-11T22:53:16.797 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: ceph --format json osd map test1 obj1-p1 2020-03-11T22:53:16.797 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: jq .acting_primary 2020-03-11T22:53:17.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: local primary=1 2020-03-11T22:53:17.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: ceph --format json osd map test1 obj1-p1 2020-03-11T22:53:17.153 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2020-03-11T22:53:17.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:147: TEST_backfill_priority: local otherosd=0 2020-03-11T22:53:17.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:149: TEST_backfill_priority: ceph pg dump pgs 2020-03-11T22:53:17.823 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:53:17.823 INFO:tasks.workunit.client.0.smithi145.stdout:7.0 38 0 0 0 0 396361728 0 0 13 13 active+clean 2020-03-11T22:52:23.408239+0000 96'113 96:164 [1] 1 [1] 1 0'0 2020-03-11T22:51:55.888484+0000 0'0 2020-03-11T22:51:55.888484+0000 0 2020-03-11T22:53:17.823 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:51:44.073807+0000 96'150 96:210 [1,0] 1 [1,0] 1 0'0 2020-03-11T22:51:42.130628+0000 0'0 2020-03-11T22:51:42.130628+0000 0 2020-03-11T22:53:17.823 INFO:tasks.workunit.client.0.smithi145.stdout:3.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:52:22.981723+0000 96'150 96:215 [1] 1 [1] 1 0'0 2020-03-11T22:51:46.721759+0000 0'0 2020-03-11T22:51:46.721759+0000 0 2020-03-11T22:53:17.823 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:53:17.823 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:53:17.824 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:53:17.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:150: TEST_backfill_priority: ERRORS=0 2020-03-11T22:53:17.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:152: TEST_backfill_priority: ceph osd set nobackfill 2020-03-11T22:53:18.495 INFO:tasks.workunit.client.0.smithi145.stderr:nobackfill is set 2020-03-11T22:53:18.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:153: TEST_backfill_priority: ceph osd set noout 2020-03-11T22:53:19.246 INFO:tasks.workunit.client.0.smithi145.stderr:noout is set 2020-03-11T22:53:19.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:157: TEST_backfill_priority: ceph osd pool set test7 size 2 2020-03-11T22:53:20.007 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 7 size to 2 2020-03-11T22:53:20.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:158: TEST_backfill_priority: sleep 2 2020-03-11T22:53:22.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:160: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:53:22.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:53:22.018 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:53:22.018 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:53:22.018 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:22.018 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56096 2020-03-11T22:53:22.019 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56096/ceph-osd.1.asok 2020-03-11T22:53:22.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:160: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:53:22.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:160: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:22.115 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:53:22.115 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:53:22.116 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:53:22.116 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:53:22.116 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:53:22.116 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:53:22.116 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:53:22.116 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2020-03-11T22:53:22.116 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2020-03-11T22:53:22.117 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:53:22.117 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:53:22.117 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:53:22.117 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:53:22.117 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:53:22.117 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:53:22.117 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:53:22.118 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:53:22.118 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:53:22.118 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:53:22.118 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:53:22.123 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:163: TEST_backfill_priority: seq 1 10 2020-03-11T22:53:22.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:163: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2020-03-11T22:53:22.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: ceph pg force-backfill 7.0 2020-03-11T22:53:22.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2020-03-11T22:53:22.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:168: TEST_backfill_priority: '[' 1 = 10 ']' 2020-03-11T22:53:22.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:172: TEST_backfill_priority: sleep 2 2020-03-11T22:53:24.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:163: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2020-03-11T22:53:24.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2020-03-11T22:53:24.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: ceph pg force-backfill 7.0 2020-03-11T22:53:24.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:166: TEST_backfill_priority: break 2020-03-11T22:53:24.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:174: TEST_backfill_priority: flush_pg_stats 2020-03-11T22:53:24.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:53:24.848 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:53:25.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:53:25.193 INFO:tasks.workunit.client.0.smithi145.stderr:1 2020-03-11T22:53:25.193 INFO:tasks.workunit.client.0.smithi145.stderr:2 2020-03-11T22:53:25.193 INFO:tasks.workunit.client.0.smithi145.stderr:3 2020-03-11T22:53:25.193 INFO:tasks.workunit.client.0.smithi145.stderr:4' 2020-03-11T22:53:25.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:53:25.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:25.193 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:53:25.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836512 2020-03-11T22:53:25.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836512' 2020-03-11T22:53:25.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:25.374 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:53:25.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672992 2020-03-11T22:53:25.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836512 1-42949672992' 2020-03-11T22:53:25.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:25.501 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:53:25.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509470 2020-03-11T22:53:25.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836512 1-42949672992 2-64424509470' 2020-03-11T22:53:25.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:25.628 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:53:25.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345948 2020-03-11T22:53:25.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836512 1-42949672992 2-64424509470 3-85899345948' 2020-03-11T22:53:25.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:25.758 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:53:25.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182427 2020-03-11T22:53:25.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836512 1-42949672992 2-64424509470 3-85899345948 4-107374182427' 2020-03-11T22:53:25.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:25.884 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836512 2020-03-11T22:53:25.885 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:25.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:53:25.885 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836512 2020-03-11T22:53:25.886 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:25.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836512 2020-03-11T22:53:25.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2020-03-11T22:53:25.887 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:53:25.887 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836512 2020-03-11T22:53:26.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836510 -lt 21474836512 2020-03-11T22:53:26.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:53:27.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:53:27.234 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:53:27.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836512 -lt 21474836512 2020-03-11T22:53:27.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:27.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672992 2020-03-11T22:53:27.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:27.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:53:27.590 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672992 2020-03-11T22:53:27.590 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:27.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672992 2020-03-11T22:53:27.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672992' 2020-03-11T22:53:27.591 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:53:27.592 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672992 2020-03-11T22:53:27.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672992 -lt 42949672992 2020-03-11T22:53:27.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:27.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509470 2020-03-11T22:53:27.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:27.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:53:27.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509470 2020-03-11T22:53:27.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:27.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509470 2020-03-11T22:53:27.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509470' 2020-03-11T22:53:27.970 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:53:27.970 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509470 2020-03-11T22:53:28.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509470 -lt 64424509470 2020-03-11T22:53:28.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:28.336 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345948 2020-03-11T22:53:28.336 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:28.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:53:28.337 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345948 2020-03-11T22:53:28.337 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:28.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345948 2020-03-11T22:53:28.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345948' 2020-03-11T22:53:28.338 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:53:28.338 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345948 2020-03-11T22:53:28.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345949 -lt 85899345948 2020-03-11T22:53:28.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:28.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182427 2020-03-11T22:53:28.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:28.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:53:28.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182427 2020-03-11T22:53:28.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:28.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182427 2020-03-11T22:53:28.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182427' 2020-03-11T22:53:28.687 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:53:28.687 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182427 2020-03-11T22:53:29.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182428 -lt 107374182427 2020-03-11T22:53:29.035 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:175: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:53:29.035 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:53:29.035 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:53:29.035 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:53:29.035 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:29.035 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56096 2020-03-11T22:53:29.036 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56096/ceph-osd.1.asok 2020-03-11T22:53:29.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:175: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:53:29.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:175: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:29.123 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:53:29.123 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:53:29.123 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:53:29.124 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:53:29.124 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:53:29.124 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:53:29.124 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:53:29.124 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2020-03-11T22:53:29.124 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 254, 2020-03-11T22:53:29.124 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:53:29.125 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:53:29.125 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:53:29.125 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:53:29.125 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:53:29.125 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:53:29.125 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:53:29.126 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:53:29.126 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:53:29.126 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:53:29.126 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:53:29.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:177: TEST_backfill_priority: ceph osd out osd.0 2020-03-11T22:53:29.641 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already out. 2020-03-11T22:53:29.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:178: TEST_backfill_priority: sleep 2 2020-03-11T22:53:31.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:179: TEST_backfill_priority: flush_pg_stats 2020-03-11T22:53:31.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:53:31.651 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:53:32.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:53:32.009 INFO:tasks.workunit.client.0.smithi145.stderr:1 2020-03-11T22:53:32.009 INFO:tasks.workunit.client.0.smithi145.stderr:2 2020-03-11T22:53:32.009 INFO:tasks.workunit.client.0.smithi145.stderr:3 2020-03-11T22:53:32.009 INFO:tasks.workunit.client.0.smithi145.stderr:4' 2020-03-11T22:53:32.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:53:32.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:32.010 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:53:32.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836515 2020-03-11T22:53:32.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836515' 2020-03-11T22:53:32.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:32.135 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:53:32.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672995 2020-03-11T22:53:32.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836515 1-42949672995' 2020-03-11T22:53:32.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:32.273 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:53:32.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509473 2020-03-11T22:53:32.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836515 1-42949672995 2-64424509473' 2020-03-11T22:53:32.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:32.401 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:53:32.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345952 2020-03-11T22:53:32.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836515 1-42949672995 2-64424509473 3-85899345952' 2020-03-11T22:53:32.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:32.527 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:53:32.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182431 2020-03-11T22:53:32.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836515 1-42949672995 2-64424509473 3-85899345952 4-107374182431' 2020-03-11T22:53:32.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:32.652 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836515 2020-03-11T22:53:32.652 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:32.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:53:32.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836515 2020-03-11T22:53:32.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:32.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836515 2020-03-11T22:53:32.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836515' 2020-03-11T22:53:32.654 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:53:32.655 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836515 2020-03-11T22:53:33.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836515 -lt 21474836515 2020-03-11T22:53:33.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:33.002 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672995 2020-03-11T22:53:33.002 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:33.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:53:33.003 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672995 2020-03-11T22:53:33.003 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:33.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672995 2020-03-11T22:53:33.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672995' 2020-03-11T22:53:33.004 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:53:33.005 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672995 2020-03-11T22:53:33.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672993 -lt 42949672995 2020-03-11T22:53:33.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:53:34.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:53:34.382 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:53:34.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672996 -lt 42949672995 2020-03-11T22:53:34.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:34.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509473 2020-03-11T22:53:34.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:34.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:53:34.737 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509473 2020-03-11T22:53:34.737 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:34.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509473 2020-03-11T22:53:34.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509473' 2020-03-11T22:53:34.738 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:53:34.738 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509473 2020-03-11T22:53:35.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509473 -lt 64424509473 2020-03-11T22:53:35.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:35.086 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345952 2020-03-11T22:53:35.086 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:35.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:53:35.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345952 2020-03-11T22:53:35.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:35.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345952 2020-03-11T22:53:35.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345952' 2020-03-11T22:53:35.088 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345952 2020-03-11T22:53:35.088 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:53:35.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345952 -lt 85899345952 2020-03-11T22:53:35.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:35.439 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182431 2020-03-11T22:53:35.439 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:35.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:53:35.440 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182431 2020-03-11T22:53:35.440 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:35.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182431 2020-03-11T22:53:35.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182431' 2020-03-11T22:53:35.441 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:53:35.441 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182431 2020-03-11T22:53:35.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182431 -lt 107374182431 2020-03-11T22:53:35.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:180: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:53:35.788 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:53:35.788 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:53:35.788 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:53:35.788 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:35.788 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56096 2020-03-11T22:53:35.788 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56096/ceph-osd.1.asok 2020-03-11T22:53:35.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:180: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:53:35.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:180: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:35.875 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:53:35.876 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:53:35.876 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:53:35.876 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:53:35.876 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2020-03-11T22:53:35.876 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:53:35.876 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2020-03-11T22:53:35.876 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:53:35.877 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:53:35.877 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:53:35.877 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:53:35.877 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:53:35.877 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:53:35.877 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:53:35.878 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:53:35.878 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:53:35.878 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:53:35.878 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:53:35.878 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2020-03-11T22:53:35.878 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 254, 2020-03-11T22:53:35.878 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:53:35.879 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:53:35.879 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:53:35.879 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:53:35.879 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:53:35.879 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:53:35.879 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:53:35.879 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:53:35.879 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:53:35.880 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:53:35.880 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:53:35.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:181: TEST_backfill_priority: ceph pg dump pgs 2020-03-11T22:53:36.203 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:53:36.203 INFO:tasks.workunit.client.0.smithi145.stdout:7.0 50 0 50 0 0 524288000 0 0 50 50 active+undersized+degraded+remapped+backfilling+forced_backfill 2020-03-11T22:53:24.743345+0000 96'150 106:220 [1,4] 1 [1] 1 0'0 2020-03-11T22:51:55.888484+0000 0'0 2020-03-11T22:51:55.888484+0000 0 2020-03-11T22:53:36.203 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 50 0 0 50 0 524288000 0 0 50 50 active+remapped+backfill_wait 2020-03-11T22:53:30.703657+0000 96'150 107:227 [1,2] 1 [1,0] 1 0'0 2020-03-11T22:51:42.130628+0000 0'0 2020-03-11T22:51:42.130628+0000 0 2020-03-11T22:53:36.203 INFO:tasks.workunit.client.0.smithi145.stdout:3.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:52:22.981723+0000 96'150 106:224 [1] 1 [1] 1 0'0 2020-03-11T22:51:46.721759+0000 0'0 2020-03-11T22:51:46.721759+0000 0 2020-03-11T22:53:36.203 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:53:36.203 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:53:36.204 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:53:36.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:183: TEST_backfill_priority: ceph osd pool set test3 size 2 2020-03-11T22:53:36.846 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 3 size to 2 2020-03-11T22:53:36.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:184: TEST_backfill_priority: sleep 2 2020-03-11T22:53:38.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:185: TEST_backfill_priority: flush_pg_stats 2020-03-11T22:53:38.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:53:38.856 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:53:39.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:53:39.215 INFO:tasks.workunit.client.0.smithi145.stderr:1 2020-03-11T22:53:39.216 INFO:tasks.workunit.client.0.smithi145.stderr:2 2020-03-11T22:53:39.216 INFO:tasks.workunit.client.0.smithi145.stderr:3 2020-03-11T22:53:39.216 INFO:tasks.workunit.client.0.smithi145.stderr:4' 2020-03-11T22:53:39.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:53:39.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:39.217 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:53:39.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836519 2020-03-11T22:53:39.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836519' 2020-03-11T22:53:39.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:39.342 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:53:39.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672999 2020-03-11T22:53:39.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836519 1-42949672999' 2020-03-11T22:53:39.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:39.470 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:53:39.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509476 2020-03-11T22:53:39.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836519 1-42949672999 2-64424509476' 2020-03-11T22:53:39.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:39.648 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:53:39.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345955 2020-03-11T22:53:39.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836519 1-42949672999 2-64424509476 3-85899345955' 2020-03-11T22:53:39.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:39.774 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:53:39.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182434 2020-03-11T22:53:39.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836519 1-42949672999 2-64424509476 3-85899345955 4-107374182434' 2020-03-11T22:53:39.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:39.902 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836519 2020-03-11T22:53:39.902 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:39.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:53:39.903 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836519 2020-03-11T22:53:39.903 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:39.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836519 2020-03-11T22:53:39.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836519' 2020-03-11T22:53:39.904 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:53:39.904 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836519 2020-03-11T22:53:40.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836517 -lt 21474836519 2020-03-11T22:53:40.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:53:41.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:53:41.252 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:53:41.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836519 -lt 21474836519 2020-03-11T22:53:41.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:41.609 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672999 2020-03-11T22:53:41.609 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:41.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:53:41.610 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672999 2020-03-11T22:53:41.610 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:41.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672999 2020-03-11T22:53:41.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672999' 2020-03-11T22:53:41.611 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:53:41.611 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672999 2020-03-11T22:53:41.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672999 -lt 42949672999 2020-03-11T22:53:41.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:41.957 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509476 2020-03-11T22:53:41.957 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:41.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:53:41.958 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509476 2020-03-11T22:53:41.958 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:41.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509476 2020-03-11T22:53:41.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509476' 2020-03-11T22:53:41.960 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:53:41.960 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509476 2020-03-11T22:53:42.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509477 -lt 64424509476 2020-03-11T22:53:42.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:42.323 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345955 2020-03-11T22:53:42.323 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:42.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:53:42.324 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345955 2020-03-11T22:53:42.324 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:42.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345955 2020-03-11T22:53:42.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345955' 2020-03-11T22:53:42.325 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:53:42.325 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345955 2020-03-11T22:53:42.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345956 -lt 85899345955 2020-03-11T22:53:42.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:42.670 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182434 2020-03-11T22:53:42.671 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:42.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:53:42.672 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182434 2020-03-11T22:53:42.672 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:42.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182434 2020-03-11T22:53:42.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182434' 2020-03-11T22:53:42.673 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182434 2020-03-11T22:53:42.673 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:53:43.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182434 -lt 107374182434 2020-03-11T22:53:43.020 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:186: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:53:43.020 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:53:43.020 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:53:43.020 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:53:43.021 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:43.021 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56096 2020-03-11T22:53:43.021 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56096/ceph-osd.1.asok 2020-03-11T22:53:43.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:186: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:53:43.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:186: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:43.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:187: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:53:43.115 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:53:43.116 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:53:43.116 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:53:43.116 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:53:43.116 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2020-03-11T22:53:43.116 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:53:43.116 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2020-03-11T22:53:43.117 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:53:43.117 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:53:43.117 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:53:43.117 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:53:43.117 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:53:43.117 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:53:43.117 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:53:43.117 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:53:43.118 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:53:43.118 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 151, 2020-03-11T22:53:43.118 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:53:43.118 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:53:43.118 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2020-03-11T22:53:43.118 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2020-03-11T22:53:43.118 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:53:43.119 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:53:43.119 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:53:43.119 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:53:43.119 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:53:43.119 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:53:43.119 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:53:43.119 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2020-03-11T22:53:43.119 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 254, 2020-03-11T22:53:43.120 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:53:43.120 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:53:43.120 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:53:43.120 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:53:43.120 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:53:43.120 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:53:43.120 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:53:43.121 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:53:43.121 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:53:43.121 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:53:43.121 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:53:43.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:188: TEST_backfill_priority: ceph pg dump pgs 2020-03-11T22:53:43.438 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:53:43.439 INFO:tasks.workunit.client.0.smithi145.stdout:7.0 50 0 50 0 0 524288000 0 0 50 50 active+undersized+degraded+remapped+backfilling+forced_backfill 2020-03-11T22:53:24.743345+0000 96'150 109:222 [1,4] 1 [1] 1 0'0 2020-03-11T22:51:55.888484+0000 0'0 2020-03-11T22:51:55.888484+0000 0 2020-03-11T22:53:43.439 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 50 0 0 50 0 524288000 0 0 50 50 active+remapped+backfill_wait 2020-03-11T22:53:30.703657+0000 96'150 109:229 [1,2] 1 [1,0] 1 0'0 2020-03-11T22:51:42.130628+0000 0'0 2020-03-11T22:51:42.130628+0000 0 2020-03-11T22:53:43.439 INFO:tasks.workunit.client.0.smithi145.stdout:3.0 50 0 0 0 0 524288000 0 0 50 50 remapped+peering 2020-03-11T22:53:39.337782+0000 96'150 110:228 [1,2] 1 [1] 1 0'0 2020-03-11T22:51:46.721759+0000 0'0 2020-03-11T22:51:46.721759+0000 0 2020-03-11T22:53:43.439 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:53:43.439 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:53:43.440 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:53:43.449 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:190: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:53:43.449 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:190: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "1.0")).prio' 2020-03-11T22:53:43.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:190: TEST_backfill_priority: PRIO=110 2020-03-11T22:53:43.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:191: TEST_backfill_priority: '[' 110 '!=' 110 ']' 2020-03-11T22:53:43.452 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:53:43.452 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: jq '.local_reservations.in_progress[0].item' 2020-03-11T22:53:43.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: eval 'ITEM="7.0"' 2020-03-11T22:53:43.454 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: ITEM=7.0 2020-03-11T22:53:43.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:199: TEST_backfill_priority: '[' 7.0 '!=' 7.0 ']' 2020-03-11T22:53:43.455 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:204: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:53:43.455 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:204: TEST_backfill_priority: jq '.local_reservations.in_progress[0].prio' 2020-03-11T22:53:43.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:204: TEST_backfill_priority: PRIO=254 2020-03-11T22:53:43.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:205: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2020-03-11T22:53:43.457 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:213: TEST_backfill_priority: seq 1 10 2020-03-11T22:53:43.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:213: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2020-03-11T22:53:43.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:215: TEST_backfill_priority: ceph pg force-backfill 3.0 2020-03-11T22:53:43.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:215: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2020-03-11T22:53:43.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:218: TEST_backfill_priority: '[' 1 = 10 ']' 2020-03-11T22:53:43.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:222: TEST_backfill_priority: sleep 2 2020-03-11T22:53:45.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:213: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2020-03-11T22:53:45.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:215: TEST_backfill_priority: ceph pg force-backfill 3.0 2020-03-11T22:53:45.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:215: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2020-03-11T22:53:46.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:216: TEST_backfill_priority: break 2020-03-11T22:53:46.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:224: TEST_backfill_priority: sleep 2 2020-03-11T22:53:48.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:225: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:53:48.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:53:48.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:53:48.144 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:53:48.144 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:48.144 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56096 2020-03-11T22:53:48.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56096/ceph-osd.1.asok 2020-03-11T22:53:48.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:225: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:53:48.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:225: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:48.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:226: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:53:48.251 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:53:48.251 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:53:48.251 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:53:48.251 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:53:48.251 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2020-03-11T22:53:48.251 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:53:48.252 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2020-03-11T22:53:48.252 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:53:48.252 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:53:48.252 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:53:48.252 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:53:48.252 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:53:48.252 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:53:48.253 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:53:48.253 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:53:48.253 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:53:48.253 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 254, 2020-03-11T22:53:48.253 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:53:48.253 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:53:48.253 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2020-03-11T22:53:48.254 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 254, 2020-03-11T22:53:48.254 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:53:48.254 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:53:48.254 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:53:48.254 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:53:48.254 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:53:48.254 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:53:48.255 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:53:48.255 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2020-03-11T22:53:48.255 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 254, 2020-03-11T22:53:48.255 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:53:48.255 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:53:48.255 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:53:48.255 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:53:48.255 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:53:48.256 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:53:48.256 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:53:48.256 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:53:48.256 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:53:48.256 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:53:48.256 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:53:48.257 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:227: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:53:48.258 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:227: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2020-03-11T22:53:48.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:227: TEST_backfill_priority: PRIO=254 2020-03-11T22:53:48.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:228: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2020-03-11T22:53:48.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:233: TEST_backfill_priority: flush_pg_stats 2020-03-11T22:53:48.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:53:48.259 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:53:48.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:53:48.603 INFO:tasks.workunit.client.0.smithi145.stderr:1 2020-03-11T22:53:48.604 INFO:tasks.workunit.client.0.smithi145.stderr:2 2020-03-11T22:53:48.604 INFO:tasks.workunit.client.0.smithi145.stderr:3 2020-03-11T22:53:48.604 INFO:tasks.workunit.client.0.smithi145.stderr:4' 2020-03-11T22:53:48.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:53:48.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:48.605 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:53:48.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836523 2020-03-11T22:53:48.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836523' 2020-03-11T22:53:48.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:48.728 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:53:48.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949673003 2020-03-11T22:53:48.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836523 1-42949673003' 2020-03-11T22:53:48.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:48.853 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:53:48.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509480 2020-03-11T22:53:48.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836523 1-42949673003 2-64424509480' 2020-03-11T22:53:48.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:48.980 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:53:49.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345959 2020-03-11T22:53:49.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836523 1-42949673003 2-64424509480 3-85899345959' 2020-03-11T22:53:49.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:49.106 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:53:49.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182438 2020-03-11T22:53:49.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836523 1-42949673003 2-64424509480 3-85899345959 4-107374182438' 2020-03-11T22:53:49.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:49.233 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836523 2020-03-11T22:53:49.233 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:49.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:53:49.235 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836523 2020-03-11T22:53:49.235 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:49.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836523 2020-03-11T22:53:49.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836523' 2020-03-11T22:53:49.236 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:53:49.236 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836523 2020-03-11T22:53:49.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836521 -lt 21474836523 2020-03-11T22:53:49.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:53:50.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:53:50.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:53:50.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836523 -lt 21474836523 2020-03-11T22:53:50.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:50.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949673003 2020-03-11T22:53:50.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:50.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:53:50.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949673003 2020-03-11T22:53:50.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:50.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949673003 2020-03-11T22:53:50.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949673003' 2020-03-11T22:53:50.970 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:53:50.970 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949673003 2020-03-11T22:53:51.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949673003 -lt 42949673003 2020-03-11T22:53:51.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:51.316 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509480 2020-03-11T22:53:51.316 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:51.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:53:51.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509480 2020-03-11T22:53:51.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:51.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509480 2020-03-11T22:53:51.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509480' 2020-03-11T22:53:51.318 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:53:51.318 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509480 2020-03-11T22:53:51.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509481 -lt 64424509480 2020-03-11T22:53:51.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:51.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345959 2020-03-11T22:53:51.687 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:51.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:53:51.687 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345959 2020-03-11T22:53:51.688 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:51.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345959 2020-03-11T22:53:51.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345959' 2020-03-11T22:53:51.689 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:53:51.689 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345959 2020-03-11T22:53:52.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345959 -lt 85899345959 2020-03-11T22:53:52.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:52.071 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182438 2020-03-11T22:53:52.071 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:52.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:53:52.072 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182438 2020-03-11T22:53:52.073 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:52.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182438 2020-03-11T22:53:52.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182438' 2020-03-11T22:53:52.074 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:53:52.074 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182438 2020-03-11T22:53:52.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182438 -lt 107374182438 2020-03-11T22:53:52.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:236: TEST_backfill_priority: ceph pg cancel-force-backfill 7.0 2020-03-11T22:53:52.752 INFO:tasks.workunit.client.0.smithi145.stderr:instructing pg(s) [7.0] on osd.1 to cancel-force-backfill; 2020-03-11T22:53:52.752 INFO:tasks.workunit.client.0.smithi145.stderr: 2020-03-11T22:53:52.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:237: TEST_backfill_priority: sleep 2 2020-03-11T22:53:54.763 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:238: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:53:54.763 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:53:54.763 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:53:54.763 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:53:54.763 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:54.763 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56096 2020-03-11T22:53:54.764 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56096/ceph-osd.1.asok 2020-03-11T22:53:54.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:238: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:53:54.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:238: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:54.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:239: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:53:54.867 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:53:54.868 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:53:54.868 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:53:54.868 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:53:54.868 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2020-03-11T22:53:54.868 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:53:54.868 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2020-03-11T22:53:54.868 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:53:54.869 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:53:54.869 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:53:54.869 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:53:54.869 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:53:54.869 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:53:54.869 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:53:54.869 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:53:54.870 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:53:54.870 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 151, 2020-03-11T22:53:54.870 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:53:54.870 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:53:54.870 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2020-03-11T22:53:54.870 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2020-03-11T22:53:54.870 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:53:54.871 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:53:54.871 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:53:54.871 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:53:54.871 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:53:54.871 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:53:54.871 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:53:54.871 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2020-03-11T22:53:54.871 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 254, 2020-03-11T22:53:54.872 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:53:54.872 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:53:54.872 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:53:54.872 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:53:54.872 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:53:54.872 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:53:54.872 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:53:54.873 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:53:54.873 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:53:54.873 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:53:54.873 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:53:54.874 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:240: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:53:54.874 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:240: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "7.0")).prio' 2020-03-11T22:53:54.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:240: TEST_backfill_priority: PRIO=151 2020-03-11T22:53:54.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:241: TEST_backfill_priority: '[' 151 '!=' 151 ']' 2020-03-11T22:53:54.875 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:53:54.875 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: jq '.local_reservations.in_progress[0].item' 2020-03-11T22:53:54.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: eval 'ITEM="3.0"' 2020-03-11T22:53:54.875 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: ITEM=3.0 2020-03-11T22:53:54.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:248: TEST_backfill_priority: '[' 3.0 '!=' 3.0 ']' 2020-03-11T22:53:54.876 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:253: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:53:54.876 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:253: TEST_backfill_priority: jq '.local_reservations.in_progress[0].prio' 2020-03-11T22:53:54.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:253: TEST_backfill_priority: PRIO=254 2020-03-11T22:53:54.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:254: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2020-03-11T22:53:54.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:261: TEST_backfill_priority: ceph pg cancel-force-backfill 3.0 2020-03-11T22:53:55.207 INFO:tasks.workunit.client.0.smithi145.stderr:instructing pg(s) [3.0] on osd.1 to cancel-force-backfill; 2020-03-11T22:53:55.208 INFO:tasks.workunit.client.0.smithi145.stderr: 2020-03-11T22:53:55.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:262: TEST_backfill_priority: sleep 5 2020-03-11T22:54:00.218 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:263: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:54:00.218 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:54:00.218 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:54:00.218 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:54:00.219 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:00.219 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56096 2020-03-11T22:54:00.219 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56096/ceph-osd.1.asok 2020-03-11T22:54:00.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:263: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:54:00.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:263: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:00.314 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:54:00.314 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:54:00.314 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:00.314 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:00.315 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2020-03-11T22:54:00.315 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:00.315 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2020-03-11T22:54:00.315 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:54:00.315 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:00.315 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:54:00.316 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:54:00.316 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:00.316 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:00.316 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:00.316 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:00.316 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:00.316 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 151, 2020-03-11T22:54:00.317 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:54:00.317 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:00.317 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2020-03-11T22:54:00.317 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2020-03-11T22:54:00.318 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:00.318 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:00.318 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:00.318 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:00.318 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:54:00.318 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:54:00.318 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:00.319 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2020-03-11T22:54:00.319 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2020-03-11T22:54:00.319 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:00.319 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:00.319 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:00.319 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:00.319 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:54:00.320 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:00.320 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:00.320 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:54:00.320 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:54:00.320 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:00.320 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:54:00.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:266: TEST_backfill_priority: flush_pg_stats 2020-03-11T22:54:00.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:54:00.322 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:54:00.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:54:00.668 INFO:tasks.workunit.client.0.smithi145.stderr:1 2020-03-11T22:54:00.669 INFO:tasks.workunit.client.0.smithi145.stderr:2 2020-03-11T22:54:00.669 INFO:tasks.workunit.client.0.smithi145.stderr:3 2020-03-11T22:54:00.669 INFO:tasks.workunit.client.0.smithi145.stderr:4' 2020-03-11T22:54:00.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:54:00.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:00.669 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:54:00.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836527 2020-03-11T22:54:00.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836527' 2020-03-11T22:54:00.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:00.795 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:54:00.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949673007 2020-03-11T22:54:00.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836527 1-42949673007' 2020-03-11T22:54:00.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:00.920 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:54:01.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509485 2020-03-11T22:54:01.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836527 1-42949673007 2-64424509485' 2020-03-11T22:54:01.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:01.047 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:54:01.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345964 2020-03-11T22:54:01.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836527 1-42949673007 2-64424509485 3-85899345964' 2020-03-11T22:54:01.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:01.173 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:54:01.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182442 2020-03-11T22:54:01.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836527 1-42949673007 2-64424509485 3-85899345964 4-107374182442' 2020-03-11T22:54:01.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:01.299 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836527 2020-03-11T22:54:01.299 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:01.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:54:01.300 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836527 2020-03-11T22:54:01.300 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:01.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836527 2020-03-11T22:54:01.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836527' 2020-03-11T22:54:01.301 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:54:01.302 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836527 2020-03-11T22:54:01.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836525 -lt 21474836527 2020-03-11T22:54:01.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:54:02.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:54:02.652 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:54:03.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836527 -lt 21474836527 2020-03-11T22:54:03.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:03.004 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949673007 2020-03-11T22:54:03.004 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:03.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:54:03.005 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949673007 2020-03-11T22:54:03.005 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:03.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949673007 2020-03-11T22:54:03.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949673007' 2020-03-11T22:54:03.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:54:03.006 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949673007 2020-03-11T22:54:03.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949673007 -lt 42949673007 2020-03-11T22:54:03.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:03.354 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509485 2020-03-11T22:54:03.354 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:03.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:54:03.355 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509485 2020-03-11T22:54:03.355 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:03.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509485 2020-03-11T22:54:03.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509485' 2020-03-11T22:54:03.356 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:54:03.356 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509485 2020-03-11T22:54:03.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509485 -lt 64424509485 2020-03-11T22:54:03.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:03.719 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345964 2020-03-11T22:54:03.719 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:03.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:54:03.720 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345964 2020-03-11T22:54:03.720 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:03.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345964 2020-03-11T22:54:03.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345964' 2020-03-11T22:54:03.721 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345964 2020-03-11T22:54:03.722 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:54:04.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345964 -lt 85899345964 2020-03-11T22:54:04.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:04.075 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182442 2020-03-11T22:54:04.076 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:04.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:54:04.077 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182442 2020-03-11T22:54:04.077 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:04.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182442 2020-03-11T22:54:04.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182442' 2020-03-11T22:54:04.078 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:54:04.078 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182442 2020-03-11T22:54:04.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182443 -lt 107374182442 2020-03-11T22:54:04.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:267: TEST_backfill_priority: ceph pg force-backfill 7.0 2020-03-11T22:54:04.783 INFO:tasks.workunit.client.0.smithi145.stderr:instructing pg(s) [7.0] on osd.1 to force-backfill; 2020-03-11T22:54:04.783 INFO:tasks.workunit.client.0.smithi145.stderr: 2020-03-11T22:54:04.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:268: TEST_backfill_priority: sleep 2 2020-03-11T22:54:06.795 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:270: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:54:06.795 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:54:06.795 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:54:06.795 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:54:06.795 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:06.796 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56096 2020-03-11T22:54:06.796 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56096/ceph-osd.1.asok 2020-03-11T22:54:06.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:270: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:54:06.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:270: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:06.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:271: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:54:06.899 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:54:06.899 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:54:06.899 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:06.899 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:06.900 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2020-03-11T22:54:06.900 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:06.900 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2020-03-11T22:54:06.900 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:54:06.900 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:06.900 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:54:06.900 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:54:06.901 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:06.901 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:06.901 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:06.901 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:06.901 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:06.901 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 151, 2020-03-11T22:54:06.901 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:54:06.902 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:06.902 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2020-03-11T22:54:06.902 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2020-03-11T22:54:06.902 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:06.902 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:06.903 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:06.903 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:06.903 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:54:06.903 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:54:06.903 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:06.903 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2020-03-11T22:54:06.903 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 254, 2020-03-11T22:54:06.904 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:06.904 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:06.904 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:06.904 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:06.904 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:54:06.904 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:06.904 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:06.904 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:54:06.905 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:54:06.905 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:06.905 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:54:06.906 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:272: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:54:06.907 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:272: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2020-03-11T22:54:06.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:272: TEST_backfill_priority: PRIO=151 2020-03-11T22:54:06.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:273: TEST_backfill_priority: '[' 151 '!=' 151 ']' 2020-03-11T22:54:06.907 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:54:06.907 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: jq '.local_reservations.in_progress[0].item' 2020-03-11T22:54:06.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: eval 'ITEM="7.0"' 2020-03-11T22:54:06.908 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: ITEM=7.0 2020-03-11T22:54:06.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:280: TEST_backfill_priority: '[' 7.0 '!=' 7.0 ']' 2020-03-11T22:54:06.908 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:285: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:54:06.908 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:285: TEST_backfill_priority: jq '.local_reservations.in_progress[0].prio' 2020-03-11T22:54:06.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:285: TEST_backfill_priority: PRIO=254 2020-03-11T22:54:06.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:286: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2020-03-11T22:54:06.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:293: TEST_backfill_priority: ceph osd unset noout 2020-03-11T22:54:07.542 INFO:tasks.workunit.client.0.smithi145.stderr:noout is unset 2020-03-11T22:54:07.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:294: TEST_backfill_priority: ceph osd unset nobackfill 2020-03-11T22:54:08.378 INFO:tasks.workunit.client.0.smithi145.stderr:nobackfill is unset 2020-03-11T22:54:08.390 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:296: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:54:08.390 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:54:08.391 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:54:08.391 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:54:08.391 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:08.391 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56096 2020-03-11T22:54:08.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56096/ceph-osd.1.asok 2020-03-11T22:54:08.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:296: TEST_backfill_priority: wait_for_clean 'CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations' 2020-03-11T22:54:08.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local 'cmd=CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations' 2020-03-11T22:54:08.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:54:08.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:54:08.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:54:08.393 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:54:08.394 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:54:08.394 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:54:08.394 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:54:08.394 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:54:08.394 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:54:08.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:54:08.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:54:08.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:54:08.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:54:08.490 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:54:08.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:54:08.823 INFO:tasks.workunit.client.0.smithi145.stderr:1 2020-03-11T22:54:08.823 INFO:tasks.workunit.client.0.smithi145.stderr:2 2020-03-11T22:54:08.823 INFO:tasks.workunit.client.0.smithi145.stderr:3 2020-03-11T22:54:08.823 INFO:tasks.workunit.client.0.smithi145.stderr:4' 2020-03-11T22:54:08.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:54:08.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:08.824 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:54:08.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836531 2020-03-11T22:54:08.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836531' 2020-03-11T22:54:08.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:08.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:54:09.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949673011 2020-03-11T22:54:09.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836531 1-42949673011' 2020-03-11T22:54:09.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:09.074 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:54:09.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509488 2020-03-11T22:54:09.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836531 1-42949673011 2-64424509488' 2020-03-11T22:54:09.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:09.200 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:54:09.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345967 2020-03-11T22:54:09.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836531 1-42949673011 2-64424509488 3-85899345967' 2020-03-11T22:54:09.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:09.326 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:54:09.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182446 2020-03-11T22:54:09.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836531 1-42949673011 2-64424509488 3-85899345967 4-107374182446' 2020-03-11T22:54:09.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:09.452 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836531 2020-03-11T22:54:09.453 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:09.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:54:09.454 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836531 2020-03-11T22:54:09.454 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:09.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836531 2020-03-11T22:54:09.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836531' 2020-03-11T22:54:09.455 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:54:09.455 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836531 2020-03-11T22:54:09.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 21474836531 2020-03-11T22:54:09.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:54:10.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:54:10.803 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:54:11.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836531 -lt 21474836531 2020-03-11T22:54:11.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:11.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949673011 2020-03-11T22:54:11.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:11.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:54:11.283 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949673011 2020-03-11T22:54:11.283 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:11.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949673011 2020-03-11T22:54:11.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949673011' 2020-03-11T22:54:11.284 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:54:11.284 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949673011 2020-03-11T22:54:11.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949673011 -lt 42949673011 2020-03-11T22:54:11.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:11.643 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509488 2020-03-11T22:54:11.644 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:11.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:54:11.644 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509488 2020-03-11T22:54:11.645 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:11.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509488 2020-03-11T22:54:11.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509488' 2020-03-11T22:54:11.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:54:11.646 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509488 2020-03-11T22:54:11.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509489 -lt 64424509488 2020-03-11T22:54:11.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:11.999 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345967 2020-03-11T22:54:12.000 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:12.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:54:12.000 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345967 2020-03-11T22:54:12.001 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:12.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345967 2020-03-11T22:54:12.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345967' 2020-03-11T22:54:12.002 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:54:12.002 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345967 2020-03-11T22:54:12.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345968 -lt 85899345967 2020-03-11T22:54:12.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:12.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182446 2020-03-11T22:54:12.527 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:12.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:54:12.527 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182446 2020-03-11T22:54:12.527 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:12.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182446 2020-03-11T22:54:12.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182446' 2020-03-11T22:54:12.529 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:54:12.529 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182446 2020-03-11T22:54:12.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182446 -lt 107374182446 2020-03-11T22:54:12.877 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:54:12.877 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:12.877 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:13.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 3 == 0 2020-03-11T22:54:13.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:13.346 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:13.346 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:13.346 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:13.346 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:13.347 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:13.347 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:13.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:13.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:13.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:13.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:14.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:54:14.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:54:14.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:54:14.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:54:14.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:14.135 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:54:14.135 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:14.222 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:54:14.222 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:54:14.222 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:14.223 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:14.223 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2020-03-11T22:54:14.223 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:14.223 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2020-03-11T22:54:14.223 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:54:14.223 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:14.224 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:54:14.224 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:54:14.224 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:14.224 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:14.224 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:14.224 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:14.224 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:14.225 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 151, 2020-03-11T22:54:14.225 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:54:14.225 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:14.225 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2020-03-11T22:54:14.225 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2020-03-11T22:54:14.225 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:14.225 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:14.226 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:14.226 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:14.226 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:54:14.226 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:54:14.226 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:14.226 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2020-03-11T22:54:14.226 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 254, 2020-03-11T22:54:14.227 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:14.227 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:14.227 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:14.227 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:14.227 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:54:14.227 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:14.227 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:14.228 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:54:14.228 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:54:14.228 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:14.228 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:54:14.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:14.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:14.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:14.332 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:14.333 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:14.333 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:14.333 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:14.333 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:14.333 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:14.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:14.669 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:14.669 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:14.669 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:15.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:54:15.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:15.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:15.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:15.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:15.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:15.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:15.176 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:15.176 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:15.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=12655260 2020-03-11T22:54:15.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 12655260 '!=' null 2020-03-11T22:54:15.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:15.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:15.626 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:54:15.626 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:15.713 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:54:15.713 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:54:15.713 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:15.713 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:15.714 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2020-03-11T22:54:15.714 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:15.714 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2020-03-11T22:54:15.714 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:54:15.714 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:15.715 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:54:15.715 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:54:15.715 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:15.715 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:15.715 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:15.716 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:15.716 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:15.716 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 151, 2020-03-11T22:54:15.716 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:54:15.717 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:15.717 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2020-03-11T22:54:15.717 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2020-03-11T22:54:15.717 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:15.717 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:15.718 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:15.718 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:15.718 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:54:15.718 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:54:15.718 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:15.719 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2020-03-11T22:54:15.719 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 254, 2020-03-11T22:54:15.719 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:15.719 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:15.719 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:15.720 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:15.720 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:54:15.720 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:15.720 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:15.721 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:54:15.721 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:54:15.721 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:15.721 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:54:15.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:15.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:15.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:15.822 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:15.822 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:15.823 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:15.823 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:15.823 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:15.823 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:16.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:16.160 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:16.160 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:16.160 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:16.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:54:16.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:16.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:16.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:16.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:16.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:16.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:16.613 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:16.613 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:17.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=10301369 2020-03-11T22:54:17.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 10301369 '!=' null 2020-03-11T22:54:17.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:17.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:17.066 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:54:17.066 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:17.162 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:54:17.162 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:54:17.162 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:17.163 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:17.163 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2020-03-11T22:54:17.163 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:17.163 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2020-03-11T22:54:17.163 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:54:17.163 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:17.164 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:54:17.164 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:54:17.164 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:17.164 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:17.164 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:17.164 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:17.164 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:17.164 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 151, 2020-03-11T22:54:17.165 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:54:17.165 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:17.165 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2020-03-11T22:54:17.165 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2020-03-11T22:54:17.165 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:17.165 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:17.167 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:17.167 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:17.167 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:54:17.167 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:54:17.168 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:17.168 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2020-03-11T22:54:17.168 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 254, 2020-03-11T22:54:17.168 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:17.168 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:17.168 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:17.168 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:17.169 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:54:17.169 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:17.169 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:17.169 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:54:17.169 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:54:17.169 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:17.169 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:54:17.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:17.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:17.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:17.264 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:17.265 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:17.265 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:17.265 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:17.265 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:17.265 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:17.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:17.604 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:17.604 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:17.605 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:18.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:54:18.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:18.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:18.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:18.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:18.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:18.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:18.052 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:18.052 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:18.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=8387155 2020-03-11T22:54:18.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 8387155 '!=' null 2020-03-11T22:54:18.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:18.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:18.513 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:54:18.514 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:18.601 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:54:18.601 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:54:18.601 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:18.601 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:18.601 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2020-03-11T22:54:18.601 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:18.602 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2020-03-11T22:54:18.602 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:54:18.602 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:18.602 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:54:18.602 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:54:18.602 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:18.602 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:18.602 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:18.603 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:18.603 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:18.603 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 151, 2020-03-11T22:54:18.603 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:54:18.603 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:18.603 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2020-03-11T22:54:18.603 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2020-03-11T22:54:18.604 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:18.604 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:18.604 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:18.604 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:18.604 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:54:18.604 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:54:18.604 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:18.604 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2020-03-11T22:54:18.605 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 254, 2020-03-11T22:54:18.605 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:18.605 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:18.605 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:18.605 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:18.605 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:54:18.606 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:18.606 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:18.606 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:54:18.606 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:54:18.606 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:18.606 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:54:18.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:18.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:18.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:18.710 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:18.710 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:18.710 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:18.710 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:18.710 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:18.711 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:19.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:19.065 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:19.065 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:19.066 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:19.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:54:19.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:19.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:19.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:19.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:19.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:19.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:19.529 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:19.530 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:20.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=8387155 2020-03-11T22:54:20.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 8387155 '!=' null 2020-03-11T22:54:20.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:20.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:20.025 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:54:20.025 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:20.110 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:54:20.111 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:54:20.111 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:20.111 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:20.111 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2020-03-11T22:54:20.111 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:20.111 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2020-03-11T22:54:20.111 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:54:20.111 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:20.112 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:54:20.112 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:54:20.112 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:20.112 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:20.112 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:20.112 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:20.112 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:20.113 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 151, 2020-03-11T22:54:20.113 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:54:20.113 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:20.113 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2020-03-11T22:54:20.113 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2020-03-11T22:54:20.113 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:20.113 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:20.114 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:20.114 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:20.114 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:54:20.114 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:54:20.114 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:20.114 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2020-03-11T22:54:20.114 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 254, 2020-03-11T22:54:20.115 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:20.115 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:20.115 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:20.115 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:20.115 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:54:20.115 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:20.115 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:20.115 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:54:20.116 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:54:20.116 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:20.116 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:54:20.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:20.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:20.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:20.219 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:20.219 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:20.220 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:20.220 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:20.220 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:20.220 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:20.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:20.561 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:20.561 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:20.562 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:21.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:54:21.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:21.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:21.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:21.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:21.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:21.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:21.015 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:21.016 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:21.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=17782621 2020-03-11T22:54:21.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 17782621 '!=' null 2020-03-11T22:54:21.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:21.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:21.601 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:54:21.601 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:21.689 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:54:21.689 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:54:21.689 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:21.689 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:21.690 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2020-03-11T22:54:21.690 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:21.690 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2020-03-11T22:54:21.690 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:54:21.690 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:21.690 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:54:21.690 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:54:21.690 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:21.691 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:21.691 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:21.691 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:21.691 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:21.691 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 151, 2020-03-11T22:54:21.691 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:54:21.691 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:21.692 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2020-03-11T22:54:21.692 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2020-03-11T22:54:21.692 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:21.692 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:21.692 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:21.692 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:21.693 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:54:21.693 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:54:21.693 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:21.693 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2020-03-11T22:54:21.693 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 254, 2020-03-11T22:54:21.693 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:21.693 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:21.693 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:21.694 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:21.694 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:54:21.694 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:21.694 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:21.694 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:54:21.694 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:54:21.694 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:21.695 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:54:21.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:21.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:21.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:21.798 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:21.798 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:21.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:21.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:21.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:21.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:22.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:22.143 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:22.143 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:22.143 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:22.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:54:22.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:22.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:22.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:22.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:22.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:22.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:22.752 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:22.752 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:23.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=17473151 2020-03-11T22:54:23.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 17473151 '!=' null 2020-03-11T22:54:23.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:23.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:23.202 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:54:23.203 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:23.290 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:54:23.290 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:54:23.290 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:23.290 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:23.291 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2020-03-11T22:54:23.291 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:23.291 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2020-03-11T22:54:23.291 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:54:23.291 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:23.291 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:54:23.291 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:54:23.291 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:23.292 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:23.292 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:23.292 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:23.292 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:23.292 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 151, 2020-03-11T22:54:23.292 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:54:23.292 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:23.292 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2020-03-11T22:54:23.293 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2020-03-11T22:54:23.293 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:23.293 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:23.293 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:23.293 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:23.293 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:54:23.293 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:54:23.294 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:23.294 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2020-03-11T22:54:23.294 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 254, 2020-03-11T22:54:23.294 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:23.294 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:23.294 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:23.294 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:23.294 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:54:23.295 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:23.295 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:23.295 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:54:23.295 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:54:23.295 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:23.295 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:54:23.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:23.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:23.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:23.399 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:23.399 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:23.399 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:23.399 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:23.400 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:23.400 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:23.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:23.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:23.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:23.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:24.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:54:24.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:24.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:24.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:24.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:24.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:24.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:24.199 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:24.200 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:24.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=28830850 2020-03-11T22:54:24.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 28830850 '!=' null 2020-03-11T22:54:24.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:24.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:24.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:54:24.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:24.742 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:54:24.742 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:54:24.742 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:24.743 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:24.743 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2020-03-11T22:54:24.743 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:24.743 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2020-03-11T22:54:24.743 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:54:24.743 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:24.743 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:54:24.744 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:54:24.744 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:24.744 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:24.744 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:24.744 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:24.744 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:24.744 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 151, 2020-03-11T22:54:24.745 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:54:24.745 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:24.745 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2020-03-11T22:54:24.745 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2020-03-11T22:54:24.745 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:24.745 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:24.745 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:24.746 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:24.746 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:54:24.746 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:54:24.746 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:24.746 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2020-03-11T22:54:24.746 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 254, 2020-03-11T22:54:24.746 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:24.746 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:24.747 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:24.747 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:24.747 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:54:24.747 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:24.747 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:24.747 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:54:24.747 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:54:24.747 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:24.748 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:54:24.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:24.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:24.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:24.851 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:24.851 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:24.852 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:24.852 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:24.852 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:24.852 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:25.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:25.192 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:25.192 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:25.192 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:25.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:54:25.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:25.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:25.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:25.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:25.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:25.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:25.643 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:25.643 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:26.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=28830850 2020-03-11T22:54:26.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 28830850 '!=' null 2020-03-11T22:54:26.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:26.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:26.263 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:54:26.263 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:26.350 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:54:26.350 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:54:26.350 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:26.350 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:26.350 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2020-03-11T22:54:26.351 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:26.351 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2020-03-11T22:54:26.351 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:54:26.351 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:26.351 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:54:26.351 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:54:26.351 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:26.352 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:26.352 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:26.352 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:26.352 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:26.352 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 151, 2020-03-11T22:54:26.352 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:54:26.352 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:26.352 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2020-03-11T22:54:26.353 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2020-03-11T22:54:26.353 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:26.353 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:26.353 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:26.353 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:26.353 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:54:26.354 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:54:26.354 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:26.354 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2020-03-11T22:54:26.354 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 254, 2020-03-11T22:54:26.354 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:26.354 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:26.355 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:26.355 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:26.355 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:54:26.355 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:26.355 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:26.355 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:54:26.355 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:54:26.356 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:26.356 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:54:26.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:26.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:26.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:26.459 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:26.459 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:26.459 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:26.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:26.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:26.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:26.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:26.800 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:26.800 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:26.800 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:27.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:54:27.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:27.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:27.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:27.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:27.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:27.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:27.257 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:27.257 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:27.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=21841455 2020-03-11T22:54:27.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 21841455 '!=' null 2020-03-11T22:54:27.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:27.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:27.759 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:54:27.759 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:27.846 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:54:27.846 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:54:27.847 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:27.847 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:27.847 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2020-03-11T22:54:27.847 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:27.847 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2020-03-11T22:54:27.847 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:54:27.847 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:27.848 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:54:27.848 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:54:27.848 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:27.848 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:27.848 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:27.848 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:27.848 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:27.848 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 151, 2020-03-11T22:54:27.849 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:54:27.849 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:27.849 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2020-03-11T22:54:27.849 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2020-03-11T22:54:27.849 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:27.849 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:27.849 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:27.850 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:27.850 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:54:27.850 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:54:27.850 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:27.850 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2020-03-11T22:54:27.850 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 254, 2020-03-11T22:54:27.850 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:27.851 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:27.851 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:27.851 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:27.851 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:54:27.851 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:27.851 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:27.851 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:54:27.851 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:54:27.852 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:27.852 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:54:27.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:27.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:27.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:27.955 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:27.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:27.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:27.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:27.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:27.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:28.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:28.294 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:28.295 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:28.295 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:28.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:54:28.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:28.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:28.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:28.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:28.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:28.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:28.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:28.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:29.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=21841373 2020-03-11T22:54:29.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 21841373 '!=' null 2020-03-11T22:54:29.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:29.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:29.221 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:54:29.221 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:29.309 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:54:29.309 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:54:29.310 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:29.310 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:29.310 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2020-03-11T22:54:29.310 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:29.310 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2020-03-11T22:54:29.311 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:54:29.311 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:29.311 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:54:29.311 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:54:29.312 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:29.312 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:29.312 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:29.313 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:29.313 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:29.313 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 151, 2020-03-11T22:54:29.313 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:54:29.314 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:29.314 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2020-03-11T22:54:29.314 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2020-03-11T22:54:29.314 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:29.315 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:29.315 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:29.315 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:29.315 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:54:29.316 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:54:29.316 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:29.316 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2020-03-11T22:54:29.316 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 254, 2020-03-11T22:54:29.316 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:29.317 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:29.317 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:29.317 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:29.317 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:54:29.317 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:29.318 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:29.318 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:54:29.318 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:54:29.318 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:29.319 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:54:29.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:29.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:29.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:29.419 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:29.419 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:29.419 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:29.419 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:29.419 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:29.419 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:29.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:29.754 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:29.755 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:29.755 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:30.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:54:30.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:30.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:30.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:30.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:30.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:30.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:30.207 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:30.207 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:30.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=33198933 2020-03-11T22:54:30.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 33198933 '!=' null 2020-03-11T22:54:30.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:30.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:30.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:54:30.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:30.822 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:54:30.823 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:54:30.823 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:30.823 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:30.823 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2020-03-11T22:54:30.824 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:30.824 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2020-03-11T22:54:30.824 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:54:30.824 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:30.825 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:54:30.825 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:54:30.825 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:30.825 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:30.825 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:30.826 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:30.826 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:54:30.826 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:54:30.826 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:30.826 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2020-03-11T22:54:30.827 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2020-03-11T22:54:30.827 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:30.827 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:30.827 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:30.828 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:30.828 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:54:30.828 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:30.828 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:30.828 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:54:30.829 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:54:30.829 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:30.829 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:54:30.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:30.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:30.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:30.932 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:30.932 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:30.932 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:30.933 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:30.933 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:30.933 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:31.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:31.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:31.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:31.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:31.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:54:31.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:31.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:31.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:31.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:31.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:31.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:31.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:31.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:32.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=33198933 2020-03-11T22:54:32.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 33198933 '!=' null 2020-03-11T22:54:32.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:32.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:32.247 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:54:32.247 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:32.337 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:54:32.337 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:54:32.337 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:32.337 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:32.337 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2020-03-11T22:54:32.337 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:32.338 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2020-03-11T22:54:32.338 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:54:32.338 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:32.338 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:54:32.338 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:54:32.338 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:32.338 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:32.339 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:32.339 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:32.339 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:54:32.339 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:54:32.339 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:32.339 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2020-03-11T22:54:32.339 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2020-03-11T22:54:32.339 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:32.340 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:32.340 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:32.340 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:32.340 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:54:32.340 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:32.340 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:32.340 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:54:32.341 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:54:32.341 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:32.341 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:54:32.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:32.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:32.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:32.447 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:32.447 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:32.447 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:32.447 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:32.447 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:32.447 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:32.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:32.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:32.786 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:32.786 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:33.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:54:33.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:33.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:33.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:33.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:33.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:33.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:33.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:33.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:33.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=17036300 2020-03-11T22:54:33.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 17036300 '!=' null 2020-03-11T22:54:33.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:33.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:33.690 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:54:33.690 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:33.777 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:54:33.777 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:54:33.777 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:33.777 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:33.777 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2020-03-11T22:54:33.777 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:33.778 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2020-03-11T22:54:33.778 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:54:33.778 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:33.778 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:54:33.778 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:54:33.778 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:33.778 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:33.779 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:33.779 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:33.779 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:54:33.779 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:54:33.779 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:33.779 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2020-03-11T22:54:33.779 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2020-03-11T22:54:33.780 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:33.780 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:33.780 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:33.780 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:33.780 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:54:33.780 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:33.780 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:33.781 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:54:33.781 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:54:33.781 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:33.781 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:54:33.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:33.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:33.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:33.886 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:33.887 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:33.887 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:33.887 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:33.887 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:33.887 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:34.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:54:34.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:34.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:34.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:34.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:54:34.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 0 2020-03-11T22:54:34.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:54:34.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=1 2020-03-11T22:54:34.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:34.678 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:54:34.678 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:34.765 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:54:34.765 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:54:34.765 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:34.765 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:34.765 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2020-03-11T22:54:34.765 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:34.766 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2020-03-11T22:54:34.766 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:54:34.766 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:34.766 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:54:34.766 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:54:34.766 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:34.766 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:34.766 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:34.767 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:34.767 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:54:34.767 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:54:34.767 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:34.767 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2020-03-11T22:54:34.767 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2020-03-11T22:54:34.767 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:34.768 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:34.768 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:34.768 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:34.768 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:54:34.768 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:34.768 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:34.769 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:54:34.769 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:54:34.769 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:34.769 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:54:34.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:34.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:34.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:34.874 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:34.874 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:34.874 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:34.874 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:34.875 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:34.875 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:35.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:54:35.213 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:35.213 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:35.213 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:35.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:54:35.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:54:35.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:35.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:35.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:35.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:35.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:35.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:35.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:36.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31451734 2020-03-11T22:54:36.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31451734 '!=' null 2020-03-11T22:54:36.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:36.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:36.134 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:54:36.134 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:36.223 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:54:36.223 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:54:36.223 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:36.223 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:36.224 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2020-03-11T22:54:36.224 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:36.224 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2020-03-11T22:54:36.224 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:54:36.224 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:36.224 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:54:36.224 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:54:36.225 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:36.225 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:36.225 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:36.225 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:36.225 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:54:36.226 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:54:36.226 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:36.226 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2020-03-11T22:54:36.226 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2020-03-11T22:54:36.226 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:36.227 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:36.227 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:36.227 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:36.227 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:54:36.227 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:36.227 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:36.228 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:54:36.228 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:54:36.228 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:36.228 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:54:36.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:36.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:36.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:36.333 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:36.333 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:36.333 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:36.333 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:36.333 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:36.334 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:36.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:54:36.671 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:36.672 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:36.672 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:37.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:54:37.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:54:37.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:37.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:37.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:37.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:37.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:37.123 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:37.124 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:37.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31451853 2020-03-11T22:54:37.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31451853 '!=' null 2020-03-11T22:54:37.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:37.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:37.572 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:54:37.572 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:37.658 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:54:37.658 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:54:37.658 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:37.658 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:37.658 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2020-03-11T22:54:37.658 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:37.659 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2020-03-11T22:54:37.659 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:54:37.659 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:37.659 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:54:37.659 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:54:37.659 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:37.659 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:37.660 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:37.660 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:37.660 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:54:37.660 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:54:37.660 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:37.660 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2020-03-11T22:54:37.660 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2020-03-11T22:54:37.660 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:37.661 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:37.661 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:37.661 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:37.661 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:54:37.661 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:37.661 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:37.661 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:54:37.661 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:54:37.662 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:37.662 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:54:37.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:37.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:37.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:37.767 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:37.768 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:37.768 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:37.768 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:37.768 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:37.768 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:38.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:54:38.109 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:38.109 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:38.109 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:38.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:54:38.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:54:38.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:38.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:38.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:38.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:38.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:38.560 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:38.560 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:39.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=14415351 2020-03-11T22:54:39.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 14415351 '!=' null 2020-03-11T22:54:39.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:39.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:39.014 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:54:39.014 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:39.101 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:54:39.101 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:54:39.101 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:39.101 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:39.101 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2020-03-11T22:54:39.102 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:39.102 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2020-03-11T22:54:39.102 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:54:39.102 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:39.102 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:54:39.102 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:54:39.103 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:39.103 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:39.103 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:39.103 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:39.103 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:54:39.103 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:54:39.104 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:39.104 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2020-03-11T22:54:39.104 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2020-03-11T22:54:39.104 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:39.105 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:39.105 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:39.105 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:39.105 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:54:39.105 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:39.105 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:39.105 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:54:39.106 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:54:39.106 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:39.106 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:54:39.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:39.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:39.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:39.210 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:39.210 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:39.210 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:39.210 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:39.211 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:39.211 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:39.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:54:39.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:39.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:39.549 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:40.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:54:40.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:54:40.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:40.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:40.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:40.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:40.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:40.001 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:40.002 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:40.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=25208162 2020-03-11T22:54:40.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 25208162 '!=' null 2020-03-11T22:54:40.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:40.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:40.559 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:54:40.560 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:40.647 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:54:40.647 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:54:40.647 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:40.647 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:40.647 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2020-03-11T22:54:40.648 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:40.648 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2020-03-11T22:54:40.648 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:54:40.648 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:40.648 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:54:40.648 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:54:40.648 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:40.649 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:40.649 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:40.649 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:40.649 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:54:40.649 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:54:40.649 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:40.649 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2020-03-11T22:54:40.649 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2020-03-11T22:54:40.650 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:40.650 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:40.650 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:40.650 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:40.650 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:54:40.650 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:40.650 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:40.651 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:54:40.651 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:54:40.651 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:40.651 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:54:40.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:40.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:40.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:40.756 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:40.756 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:40.756 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:40.756 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:40.756 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:40.756 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:41.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:54:41.093 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:41.093 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:41.094 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:41.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:54:41.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:54:41.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:41.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:41.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:41.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:41.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:41.542 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:41.542 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:41.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=25208162 2020-03-11T22:54:41.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 25208162 '!=' null 2020-03-11T22:54:41.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:41.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:41.990 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:54:41.991 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:42.080 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:54:42.080 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:54:42.080 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:42.081 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:42.081 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2020-03-11T22:54:42.081 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:42.081 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2020-03-11T22:54:42.081 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:54:42.081 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:42.081 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:54:42.081 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:54:42.082 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:42.082 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:42.082 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:42.082 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:42.082 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:54:42.082 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:54:42.082 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:42.082 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2020-03-11T22:54:42.083 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2020-03-11T22:54:42.083 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:42.083 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:42.083 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:42.083 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:42.083 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:54:42.083 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:42.084 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:42.084 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:54:42.084 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:54:42.084 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:42.084 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:54:42.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:42.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:42.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:42.189 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:42.190 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:42.190 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:42.190 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:42.190 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:42.190 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:42.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:54:42.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:42.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:42.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:42.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:54:42.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:54:42.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:42.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:42.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:42.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:42.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:42.988 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:42.988 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:43.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=21885186 2020-03-11T22:54:43.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 21885186 '!=' null 2020-03-11T22:54:43.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:43.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:43.449 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:54:43.449 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:43.535 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:54:43.535 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:54:43.536 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:43.536 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:43.536 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2020-03-11T22:54:43.536 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:43.536 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2020-03-11T22:54:43.536 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:54:43.536 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:43.537 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:54:43.537 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:54:43.537 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:43.537 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:43.537 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:43.537 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:43.537 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:54:43.538 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:54:43.538 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:43.538 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2020-03-11T22:54:43.538 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2020-03-11T22:54:43.538 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:43.538 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:43.538 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:43.539 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:43.539 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:54:43.539 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:43.539 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:43.539 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:54:43.539 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:54:43.539 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:43.540 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:54:43.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:43.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:43.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:43.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:43.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:43.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:43.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:43.647 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:43.647 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:43.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:54:43.984 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:43.984 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:43.984 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:44.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:54:44.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:54:44.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:44.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:44.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:44.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:44.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:44.546 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:44.546 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:45.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=32324939 2020-03-11T22:54:45.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 32324939 '!=' null 2020-03-11T22:54:45.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:45.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:45.013 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:54:45.013 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:45.099 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:54:45.100 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:54:45.100 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:45.100 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:45.100 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2020-03-11T22:54:45.100 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:45.101 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2020-03-11T22:54:45.101 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:54:45.101 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:45.101 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:54:45.101 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:54:45.101 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:45.101 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:45.102 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:45.102 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:45.102 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:54:45.102 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:54:45.102 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:45.102 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2020-03-11T22:54:45.102 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2020-03-11T22:54:45.102 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:45.103 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:45.103 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:45.103 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:45.103 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:54:45.103 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:45.103 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:45.103 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:54:45.103 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:54:45.104 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:45.104 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:54:45.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:45.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:45.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:45.209 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:45.209 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:45.209 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:45.209 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:45.209 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:45.209 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:45.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:54:45.550 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:45.550 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:45.550 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:46.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:54:46.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:54:46.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:46.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:46.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:46.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:46.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:46.005 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:46.005 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:46.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=22714839 2020-03-11T22:54:46.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 22714839 '!=' null 2020-03-11T22:54:46.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:46.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:46.473 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:54:46.473 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:46.560 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:54:46.561 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:54:46.561 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:46.561 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:46.561 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2020-03-11T22:54:46.561 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:46.561 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2020-03-11T22:54:46.561 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:54:46.562 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:46.562 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:54:46.562 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:54:46.562 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:46.562 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:46.562 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:46.562 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:46.563 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:54:46.563 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:54:46.563 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:46.563 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2020-03-11T22:54:46.563 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2020-03-11T22:54:46.563 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:46.563 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:46.564 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:46.564 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:46.564 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:54:46.564 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:46.564 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:46.564 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:54:46.564 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:54:46.565 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:46.565 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:54:46.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:46.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:46.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:46.670 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:46.670 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:46.670 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:46.670 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:46.670 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:46.671 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:47.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:54:47.009 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:47.009 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:47.009 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:47.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:54:47.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:54:47.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:47.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:47.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:47.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:47.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:47.467 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:47.467 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:47.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=22714839 2020-03-11T22:54:47.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 22714839 '!=' null 2020-03-11T22:54:47.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:47.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:47.927 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:54:47.927 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:48.013 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:54:48.013 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:54:48.013 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:48.014 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:48.014 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2020-03-11T22:54:48.014 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:48.014 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2020-03-11T22:54:48.014 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:54:48.014 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:48.014 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:54:48.015 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:54:48.015 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:48.015 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:48.015 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:48.015 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:48.015 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:54:48.015 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:54:48.015 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:48.016 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2020-03-11T22:54:48.016 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2020-03-11T22:54:48.016 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:48.016 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:48.016 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:48.016 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:48.016 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:54:48.016 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:48.017 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:48.017 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:54:48.017 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:54:48.017 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:48.017 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:54:48.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:48.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:48.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:48.123 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:48.123 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:48.123 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:48.123 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:48.124 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:48.124 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:48.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:54:48.459 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:48.459 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:48.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:48.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:54:48.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:54:48.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:48.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:48.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:48.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:48.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:48.917 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:48.918 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:49.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=22714800 2020-03-11T22:54:49.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 22714800 '!=' null 2020-03-11T22:54:49.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:49.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:49.373 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:54:49.373 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:49.461 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:54:49.461 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:54:49.461 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:49.462 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:49.462 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2020-03-11T22:54:49.462 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:49.462 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2020-03-11T22:54:49.462 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2020-03-11T22:54:49.462 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:49.462 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:54:49.463 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:54:49.463 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:49.463 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:49.463 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:49.463 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:49.463 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2020-03-11T22:54:49.463 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:54:49.463 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:49.464 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2020-03-11T22:54:49.464 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2020-03-11T22:54:49.464 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:49.464 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:49.464 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:49.464 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:49.464 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:54:49.465 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:49.465 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:49.465 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:54:49.465 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:54:49.465 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:49.465 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:54:49.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:49.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:49.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:49.571 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:49.571 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:49.571 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:49.571 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:49.571 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:49.571 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:49.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:54:49.907 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:49.908 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:49.908 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:50.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:54:50.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:54:50.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:50.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:50.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:50.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:50.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:50.365 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:50.366 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:50.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=33198695 2020-03-11T22:54:50.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 33198695 '!=' null 2020-03-11T22:54:50.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:50.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:50.882 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:54:50.882 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:50.970 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:54:50.970 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:54:50.971 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:50.971 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:50.971 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:54:50.971 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:54:50.971 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:50.971 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:54:50.971 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:54:50.972 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:50.972 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:50.972 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:50.972 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:50.972 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:54:50.972 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:50.972 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:50.972 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:54:50.973 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:54:50.973 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:50.973 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:54:50.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:51.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:51.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:51.085 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:51.086 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:51.086 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:51.086 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:51.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:51.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:51.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:54:51.509 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:51.509 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:51.509 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:51.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:54:51.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:54:51.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:51.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:51.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:51.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:51.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:51.963 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:51.963 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:52.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=15725681 2020-03-11T22:54:52.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 15725681 '!=' null 2020-03-11T22:54:52.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:52.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:52.539 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:54:52.539 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:52.626 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:54:52.627 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:54:52.627 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:52.627 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:52.627 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:54:52.627 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:54:52.627 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:52.628 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:54:52.628 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:54:52.628 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:52.628 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:52.628 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:52.628 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:52.629 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:54:52.629 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:52.629 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:52.629 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:54:52.629 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:54:52.629 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:52.629 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:54:52.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:52.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:52.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:52.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:52.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:52.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:52.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:52.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:52.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:53.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:54:53.074 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:53.074 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:53.074 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:53.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:54:53.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:54:53.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:53.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:53.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:53.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:53.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:53.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:53.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:54.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=15725681 2020-03-11T22:54:54.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 15725681 '!=' null 2020-03-11T22:54:54.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:54.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:54.070 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:54:54.070 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:54.156 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:54:54.156 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:54:54.156 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:54.156 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:54.156 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:54:54.157 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:54:54.157 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:54.157 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:54:54.157 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:54:54.157 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:54.158 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:54.158 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:54.158 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:54.158 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:54:54.159 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:54.159 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:54.159 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:54:54.159 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:54:54.159 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:54.159 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:54:54.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:54.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:54.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:54.265 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:54.265 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:54.265 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:54.265 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:54.266 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:54.266 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:54.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:54:54.606 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:54.607 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:54.607 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:55.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:54:55.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 1 2020-03-11T22:54:55.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:54:55.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=2 2020-03-11T22:54:55.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:55.058 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:54:55.058 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:55.149 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:54:55.149 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:54:55.150 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:55.150 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:55.150 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:54:55.150 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:54:55.150 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:55.150 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:54:55.150 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:54:55.151 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:55.151 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:55.151 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:55.151 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:55.151 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:54:55.151 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:55.151 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:55.151 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:54:55.152 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:54:55.152 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:55.152 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:54:55.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:55.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:55.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:55.258 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:55.259 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:55.259 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:55.259 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:55.259 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:55.259 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:55.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:54:55.594 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:55.594 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:55.594 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:56.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:54:56.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:54:56.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:56.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:56.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:56.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:56.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:56.047 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:56.047 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:56.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31451127 2020-03-11T22:54:56.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31451127 '!=' null 2020-03-11T22:54:56.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:56.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:56.613 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:54:56.613 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:56.700 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:54:56.700 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:54:56.700 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:56.700 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:56.700 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:54:56.700 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:54:56.701 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:56.701 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:54:56.701 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:54:56.701 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:56.701 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:56.701 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:56.701 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:56.702 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:54:56.702 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:56.702 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:56.702 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:54:56.702 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:54:56.702 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:56.702 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:54:56.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:56.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:56.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:56.810 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:56.810 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:56.810 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:56.810 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:56.810 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:56.810 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:57.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:54:57.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:57.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:57.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:57.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:54:57.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:54:57.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:57.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:57.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:57.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:57.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:57.606 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:57.606 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:58.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31451127 2020-03-11T22:54:58.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31451127 '!=' null 2020-03-11T22:54:58.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:58.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:58.061 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:54:58.061 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:58.149 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:54:58.149 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:54:58.149 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:58.150 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:58.150 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:54:58.150 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:54:58.150 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:58.150 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:54:58.150 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:54:58.150 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:58.151 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:58.151 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:58.151 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:58.151 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:54:58.151 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:58.151 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:58.151 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:54:58.152 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:54:58.152 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:58.152 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:54:58.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:58.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:58.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:58.258 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:58.258 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:58.259 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:58.259 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:58.259 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:58.259 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:58.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:54:58.599 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:58.600 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:58.600 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:59.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:54:59.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:54:59.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:59.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:59.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:59.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:59.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:59.053 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:59.053 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:59.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=15725177 2020-03-11T22:54:59.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 15725177 '!=' null 2020-03-11T22:54:59.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:59.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:59.504 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:54:59.504 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:59.591 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:54:59.591 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:54:59.591 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:59.591 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:59.591 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:54:59.592 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:54:59.592 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:54:59.592 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:54:59.592 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:54:59.592 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:54:59.592 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:59.592 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:54:59.593 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:54:59.593 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:54:59.593 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:54:59.593 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:54:59.593 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:54:59.593 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:54:59.593 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:54:59.594 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:54:59.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:59.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:59.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:59.700 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:59.701 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:59.701 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:59.701 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:59.701 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:59.701 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:00.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:55:00.039 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:00.040 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:00.040 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:00.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:55:00.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:55:00.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:00.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:00.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:00.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:00.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:00.542 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:00.542 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:00.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=26082491 2020-03-11T22:55:00.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 26082491 '!=' null 2020-03-11T22:55:00.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:00.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:00.990 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:55:00.990 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:01.077 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:55:01.077 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:55:01.077 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:55:01.077 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:55:01.078 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:55:01.078 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:55:01.078 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:01.078 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:55:01.078 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:55:01.078 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:55:01.078 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:01.079 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:55:01.079 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:01.079 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:55:01.079 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:55:01.079 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:55:01.079 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:55:01.079 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:55:01.080 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:01.080 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:55:01.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:01.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:01.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:01.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:01.187 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:01.187 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:01.187 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:01.187 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:01.187 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:01.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:55:01.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:01.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:01.583 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:02.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:55:02.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:55:02.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:02.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:02.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:02.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:02.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:02.068 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:02.069 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:02.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=22142995 2020-03-11T22:55:02.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 22142995 '!=' null 2020-03-11T22:55:02.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:02.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:02.532 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:55:02.532 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:02.618 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:55:02.618 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:55:02.618 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:55:02.619 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:55:02.619 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:55:02.619 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:55:02.619 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:02.619 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:55:02.619 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:55:02.619 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:55:02.620 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:02.620 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:55:02.620 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:02.620 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:55:02.620 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:55:02.620 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:55:02.620 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:55:02.620 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:55:02.621 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:02.621 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:55:02.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:02.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:02.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:02.727 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:02.727 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:02.727 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:02.727 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:02.728 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:02.728 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:03.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:55:03.066 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:03.066 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:03.067 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:03.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:55:03.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:55:03.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:03.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:03.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:03.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:03.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:03.522 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:03.522 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:03.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=22142995 2020-03-11T22:55:03.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 22142995 '!=' null 2020-03-11T22:55:03.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:03.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:03.975 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:55:03.975 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:04.066 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:55:04.066 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:55:04.066 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:55:04.066 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:55:04.066 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:55:04.066 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:55:04.067 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:04.067 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:55:04.067 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:55:04.067 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:55:04.067 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:04.067 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:55:04.068 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:04.068 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:55:04.068 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:55:04.068 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:55:04.068 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:55:04.068 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:55:04.068 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:04.069 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:55:04.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:04.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:04.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:04.175 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:04.175 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:04.176 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:04.176 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:04.176 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:04.176 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:04.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:55:04.513 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:04.514 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:04.514 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:04.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:55:04.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:55:04.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:04.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:04.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:04.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:04.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:04.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:04.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:05.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=32324934 2020-03-11T22:55:05.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 32324934 '!=' null 2020-03-11T22:55:05.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:05.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:05.419 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:55:05.419 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:05.507 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:55:05.507 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:55:05.507 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:55:05.507 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:55:05.507 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:55:05.507 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:55:05.507 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:05.508 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:55:05.508 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:55:05.508 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:55:05.508 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:05.508 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:55:05.508 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:05.508 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:55:05.509 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:55:05.509 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:55:05.509 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:55:05.509 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:55:05.509 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:05.509 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:55:05.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:05.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:05.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:05.616 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:05.617 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:05.617 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:05.617 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:05.617 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:05.617 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:05.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:55:05.958 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:05.958 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:05.958 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:06.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:55:06.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:55:06.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:06.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:06.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:06.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:06.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:06.438 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:06.438 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:06.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=21841292 2020-03-11T22:55:06.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 21841292 '!=' null 2020-03-11T22:55:06.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:06.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:06.888 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:55:06.888 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:06.976 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:55:06.976 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:55:06.976 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:55:06.976 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:55:06.976 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:55:06.976 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:55:06.977 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:06.977 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:55:06.977 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:55:06.977 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:55:06.977 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:06.977 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:55:06.977 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:06.978 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:55:06.978 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:55:06.978 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:55:06.978 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:55:06.978 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:55:06.978 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:06.979 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:55:06.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:07.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:07.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:07.085 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:07.085 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:07.086 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:07.086 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:07.086 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:07.086 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:07.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:55:07.422 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:07.422 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:07.422 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:07.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:55:07.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:55:07.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:07.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:07.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:07.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:07.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:07.877 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:07.878 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:08.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=21841292 2020-03-11T22:55:08.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 21841292 '!=' null 2020-03-11T22:55:08.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:08.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:08.351 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:55:08.351 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:08.437 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:55:08.438 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:55:08.438 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:55:08.438 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:55:08.438 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:55:08.438 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:55:08.438 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:08.439 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:55:08.439 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:55:08.439 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:55:08.439 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:08.439 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:55:08.439 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:08.439 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:55:08.439 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:55:08.440 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:55:08.440 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:55:08.440 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:55:08.440 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:08.440 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:55:08.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:08.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:08.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:08.547 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:08.547 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:08.547 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:08.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:08.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:08.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:08.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:55:08.885 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:08.885 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:08.885 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:09.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:55:09.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:55:09.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:09.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:09.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:09.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:09.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:09.335 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:09.335 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:09.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=21841377 2020-03-11T22:55:09.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 21841377 '!=' null 2020-03-11T22:55:09.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:09.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:09.786 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:55:09.786 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:09.874 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:55:09.874 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:55:09.874 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:55:09.875 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:55:09.875 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:55:09.875 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2020-03-11T22:55:09.875 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:09.875 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2020-03-11T22:55:09.876 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2020-03-11T22:55:09.876 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2020-03-11T22:55:09.876 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:09.876 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:55:09.877 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:09.877 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:55:09.877 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:55:09.877 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:55:09.877 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:55:09.878 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:55:09.878 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:09.878 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:55:09.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:09.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:09.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:09.984 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:09.984 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:09.984 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:09.984 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:09.984 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:09.984 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:10.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:55:10.322 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:10.323 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:10.323 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:10.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:55:10.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:55:10.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:10.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:10.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:10.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:10.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:10.778 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:10.778 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:11.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=33199320 2020-03-11T22:55:11.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 33199320 '!=' null 2020-03-11T22:55:11.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:11.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:11.277 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:55:11.278 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:11.364 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:55:11.365 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:55:11.365 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:55:11.365 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:55:11.365 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:55:11.365 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:55:11.366 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:11.366 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:55:11.366 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:55:11.366 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:55:11.366 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:55:11.366 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:55:11.367 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:11.367 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:55:11.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:11.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:11.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:11.474 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:11.475 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:11.475 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:11.475 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:11.475 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:11.475 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:11.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:55:11.807 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:11.807 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:11.807 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:12.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:55:12.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:55:12.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:12.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:12.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:12.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:12.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:12.266 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:12.267 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:12.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=17036480 2020-03-11T22:55:12.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 17036480 '!=' null 2020-03-11T22:55:12.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:12.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:12.722 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:55:12.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:12.809 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:55:12.809 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:55:12.809 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:55:12.809 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:55:12.809 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:55:12.810 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:55:12.810 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:12.810 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:55:12.810 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:55:12.810 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:55:12.810 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:55:12.810 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:55:12.811 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:12.811 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:55:12.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:12.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:12.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:12.918 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:12.918 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:12.918 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:12.918 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:12.919 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:12.919 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:13.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:55:13.250 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:13.250 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:13.250 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:13.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:55:13.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:55:13.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:13.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:13.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:13.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:13.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:13.700 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:13.701 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:14.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=17036480 2020-03-11T22:55:14.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 17036480 '!=' null 2020-03-11T22:55:14.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:14.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:14.146 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:55:14.146 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:14.234 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:55:14.234 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2020-03-11T22:55:14.234 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:55:14.234 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:55:14.235 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:55:14.235 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:55:14.235 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:14.235 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2020-03-11T22:55:14.235 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2020-03-11T22:55:14.236 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2020-03-11T22:55:14.236 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2020-03-11T22:55:14.236 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2020-03-11T22:55:14.236 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:14.236 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:55:14.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:14.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:14.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:14.343 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:14.343 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:14.343 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:14.343 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:14.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:14.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:14.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=3 2020-03-11T22:55:14.678 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:14.678 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:14.678 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:15.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 3 = 3 2020-03-11T22:55:15.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:55:15.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:55:15.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:298: TEST_backfill_priority: ceph pg dump pgs 2020-03-11T22:55:15.454 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:55:15.455 INFO:tasks.workunit.client.0.smithi145.stdout:7.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:54:30.868921+0000 96'150 120:346 [1,4] 1 [1,4] 1 0'0 2020-03-11T22:51:55.888484+0000 0'0 2020-03-11T22:51:55.888484+0000 0 2020-03-11T22:55:15.455 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:55:11.142460+0000 96'150 121:350 [1,2] 1 [1,2] 1 0'0 2020-03-11T22:51:42.130628+0000 0'0 2020-03-11T22:51:42.130628+0000 0 2020-03-11T22:55:15.455 INFO:tasks.workunit.client.0.smithi145.stdout:3.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:54:50.969092+0000 96'150 120:359 [1,2] 1 [1,2] 1 0'0 2020-03-11T22:51:46.721759+0000 0'0 2020-03-11T22:51:46.721759+0000 0 2020-03-11T22:55:15.455 INFO:tasks.workunit.client.0.smithi145.stdout: 2020-03-11T22:55:15.455 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:55:15.456 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2020-03-11T22:55:15.465 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:300: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:55:15.466 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:55:15.466 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:55:15.466 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:55:15.466 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:15.466 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56096 2020-03-11T22:55:15.466 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56096/ceph-osd.1.asok 2020-03-11T22:55:15.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:300: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:55:15.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:300: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56096/ceph-osd.1.asok dump_pgstate_history 2020-03-11T22:55:15.555 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2020-03-11T22:55:15.555 INFO:tasks.workunit.client.0.smithi145.stdout: "pgs": [ 2020-03-11T22:55:15.556 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.556 INFO:tasks.workunit.client.0.smithi145.stdout: "pg": "7.0", 2020-03-11T22:55:15.556 INFO:tasks.workunit.client.0.smithi145.stdout: "currently": "Started/Primary/Active/Clean", 2020-03-11T22:55:15.556 INFO:tasks.workunit.client.0.smithi145.stdout: "history": [ 2020-03-11T22:55:15.556 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.556 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "64", 2020-03-11T22:55:15.556 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2020-03-11T22:55:15.557 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.557 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Start", 2020-03-11T22:55:15.557 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:56.139605+0000", 2020-03-11T22:55:15.557 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:56.139616+0000" 2020-03-11T22:55:15.557 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.557 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.558 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:55:15.558 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:56.139622+0000", 2020-03-11T22:55:15.558 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:56.239814+0000" 2020-03-11T22:55:15.558 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.558 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.558 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:55:15.558 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:56.239816+0000", 2020-03-11T22:55:15.558 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:56.239880+0000" 2020-03-11T22:55:15.559 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.559 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.559 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:55:15.559 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:56.239881+0000", 2020-03-11T22:55:15.559 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:56.239895+0000" 2020-03-11T22:55:15.559 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.559 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.560 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:55:15.560 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:56.239896+0000", 2020-03-11T22:55:15.560 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:56.420856+0000" 2020-03-11T22:55:15.560 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.560 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.560 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:55:15.561 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:56.139619+0000", 2020-03-11T22:55:15.561 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:56.420866+0000" 2020-03-11T22:55:15.561 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.561 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.561 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:55:15.561 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:56.421077+0000", 2020-03-11T22:55:15.562 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:57.065127+0000" 2020-03-11T22:55:15.562 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.562 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.562 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:55:15.562 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:57.065127+0000", 2020-03-11T22:55:15.562 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:57.065212+0000" 2020-03-11T22:55:15.562 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.563 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.563 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:55:15.563 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:57.065213+0000", 2020-03-11T22:55:15.563 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:57.613234+0000" 2020-03-11T22:55:15.563 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.563 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.563 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active", 2020-03-11T22:55:15.564 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:56.420866+0000", 2020-03-11T22:55:15.564 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:57.613265+0000" 2020-03-11T22:55:15.564 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.564 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.564 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary", 2020-03-11T22:55:15.564 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:56.139617+0000", 2020-03-11T22:55:15.564 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:57.613289+0000" 2020-03-11T22:55:15.565 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.565 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.565 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started", 2020-03-11T22:55:15.565 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:56.139602+0000", 2020-03-11T22:55:15.565 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:57.613297+0000" 2020-03-11T22:55:15.566 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:15.566 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:55:15.566 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.566 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.566 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "64", 2020-03-11T22:55:15.566 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2020-03-11T22:55:15.566 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.566 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2020-03-11T22:55:15.567 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:57.613297+0000", 2020-03-11T22:55:15.567 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:57.613631+0000" 2020-03-11T22:55:15.567 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:15.567 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:55:15.567 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.567 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.567 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "94", 2020-03-11T22:55:15.567 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2020-03-11T22:55:15.568 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.568 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Start", 2020-03-11T22:55:15.568 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:57.613638+0000", 2020-03-11T22:55:15.568 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:57.613664+0000" 2020-03-11T22:55:15.568 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.568 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.568 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:55:15.569 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:57.613680+0000", 2020-03-11T22:55:15.569 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:57.707654+0000" 2020-03-11T22:55:15.569 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.569 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.569 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:55:15.569 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:57.707655+0000", 2020-03-11T22:55:15.570 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:57.707714+0000" 2020-03-11T22:55:15.570 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.570 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.570 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:55:15.570 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:57.707714+0000", 2020-03-11T22:55:15.570 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:57.707725+0000" 2020-03-11T22:55:15.570 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.571 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.571 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:55:15.571 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:57.707725+0000", 2020-03-11T22:55:15.571 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:57.914251+0000" 2020-03-11T22:55:15.571 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.571 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.571 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:55:15.571 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:57.613674+0000", 2020-03-11T22:55:15.571 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:57.914259+0000" 2020-03-11T22:55:15.572 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.572 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.572 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:55:15.572 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:57.914397+0000", 2020-03-11T22:55:15.572 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:58.107412+0000" 2020-03-11T22:55:15.572 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.572 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.572 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:55:15.573 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:58.107413+0000", 2020-03-11T22:55:15.573 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:58.107462+0000" 2020-03-11T22:55:15.573 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.573 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.573 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:55:15.573 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:58.107462+0000", 2020-03-11T22:55:15.574 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:52:23.144244+0000" 2020-03-11T22:55:15.574 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.574 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.574 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active", 2020-03-11T22:55:15.574 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:57.914260+0000", 2020-03-11T22:55:15.574 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:52:23.144262+0000" 2020-03-11T22:55:15.574 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.574 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.575 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary", 2020-03-11T22:55:15.575 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:57.613665+0000", 2020-03-11T22:55:15.575 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:52:23.144278+0000" 2020-03-11T22:55:15.575 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.575 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.575 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started", 2020-03-11T22:55:15.575 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:57.613632+0000", 2020-03-11T22:55:15.575 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:52:23.144282+0000" 2020-03-11T22:55:15.576 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:15.576 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:55:15.576 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.576 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.576 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "94", 2020-03-11T22:55:15.576 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2020-03-11T22:55:15.576 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.576 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2020-03-11T22:55:15.577 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:52:23.144282+0000", 2020-03-11T22:55:15.577 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:52:23.144468+0000" 2020-03-11T22:55:15.577 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:15.577 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:55:15.577 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.577 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.577 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "101", 2020-03-11T22:55:15.578 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2020-03-11T22:55:15.578 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.578 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Start", 2020-03-11T22:55:15.578 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:52:23.144472+0000", 2020-03-11T22:55:15.578 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:52:23.144484+0000" 2020-03-11T22:55:15.578 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.578 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.578 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:55:15.579 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:52:23.144492+0000", 2020-03-11T22:55:15.579 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:52:23.144593+0000" 2020-03-11T22:55:15.579 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.579 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.579 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:55:15.579 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:52:23.144594+0000", 2020-03-11T22:55:15.579 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:52:23.144644+0000" 2020-03-11T22:55:15.580 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.580 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.580 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:55:15.580 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:52:23.144644+0000", 2020-03-11T22:55:15.580 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:52:23.144663+0000" 2020-03-11T22:55:15.580 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.580 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.581 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:55:15.581 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:52:23.144663+0000", 2020-03-11T22:55:15.581 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:52:23.382835+0000" 2020-03-11T22:55:15.581 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.581 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.581 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:55:15.581 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:52:23.144489+0000", 2020-03-11T22:55:15.582 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:52:23.382851+0000" 2020-03-11T22:55:15.582 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.582 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.582 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:55:15.582 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:52:23.382973+0000", 2020-03-11T22:55:15.582 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:52:23.408184+0000" 2020-03-11T22:55:15.582 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.582 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.583 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:55:15.583 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:52:23.408184+0000", 2020-03-11T22:55:15.583 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:52:23.408233+0000" 2020-03-11T22:55:15.583 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.584 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.584 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:55:15.584 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:52:23.408234+0000", 2020-03-11T22:55:15.584 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:19.826277+0000" 2020-03-11T22:55:15.584 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.584 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.585 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active", 2020-03-11T22:55:15.585 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:52:23.382852+0000", 2020-03-11T22:55:15.585 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:19.826309+0000" 2020-03-11T22:55:15.585 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.585 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.585 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary", 2020-03-11T22:55:15.585 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:52:23.144485+0000", 2020-03-11T22:55:15.586 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:19.826356+0000" 2020-03-11T22:55:15.586 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.586 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.586 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started", 2020-03-11T22:55:15.586 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:52:23.144468+0000", 2020-03-11T22:55:15.586 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:19.826361+0000" 2020-03-11T22:55:15.586 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:15.587 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:55:15.587 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.587 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.587 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "101", 2020-03-11T22:55:15.587 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2020-03-11T22:55:15.587 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.587 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2020-03-11T22:55:15.588 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:19.826361+0000", 2020-03-11T22:55:15.588 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:19.826678+0000" 2020-03-11T22:55:15.588 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:15.588 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:55:15.588 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.588 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.588 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "102", 2020-03-11T22:55:15.588 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2020-03-11T22:55:15.589 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.589 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Start", 2020-03-11T22:55:15.589 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:19.826682+0000", 2020-03-11T22:55:15.589 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:19.826694+0000" 2020-03-11T22:55:15.589 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.589 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.590 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:55:15.590 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:19.826702+0000", 2020-03-11T22:55:15.590 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:19.844489+0000" 2020-03-11T22:55:15.590 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.590 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.590 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:55:15.590 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:19.844489+0000", 2020-03-11T22:55:15.591 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:19.844547+0000" 2020-03-11T22:55:15.591 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.591 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.591 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:55:15.591 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:19.826698+0000", 2020-03-11T22:55:15.591 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:19.844557+0000" 2020-03-11T22:55:15.591 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.591 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.591 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/WaitActingChange", 2020-03-11T22:55:15.592 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:19.844557+0000", 2020-03-11T22:55:15.592 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:20.006814+0000" 2020-03-11T22:55:15.592 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.592 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.592 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary", 2020-03-11T22:55:15.592 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:19.826694+0000", 2020-03-11T22:55:15.592 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:20.006833+0000" 2020-03-11T22:55:15.593 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.593 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.593 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started", 2020-03-11T22:55:15.593 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:19.826678+0000", 2020-03-11T22:55:15.593 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:20.006839+0000" 2020-03-11T22:55:15.593 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:15.593 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:55:15.594 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.594 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.594 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "102", 2020-03-11T22:55:15.594 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2020-03-11T22:55:15.594 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.594 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2020-03-11T22:55:15.594 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:20.006839+0000", 2020-03-11T22:55:15.595 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:20.007078+0000" 2020-03-11T22:55:15.595 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:15.595 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:55:15.595 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.595 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.595 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "116", 2020-03-11T22:55:15.595 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2020-03-11T22:55:15.595 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.596 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Start", 2020-03-11T22:55:15.596 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:20.007083+0000", 2020-03-11T22:55:15.596 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:20.007098+0000" 2020-03-11T22:55:15.596 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.596 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.596 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:55:15.596 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:20.007107+0000", 2020-03-11T22:55:15.596 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:20.044829+0000" 2020-03-11T22:55:15.597 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.597 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.597 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:55:15.597 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:20.044830+0000", 2020-03-11T22:55:15.597 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:20.044907+0000" 2020-03-11T22:55:15.597 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.597 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.598 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:55:15.598 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:20.044908+0000", 2020-03-11T22:55:15.598 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:20.044925+0000" 2020-03-11T22:55:15.598 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.598 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.598 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:55:15.598 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:20.044926+0000", 2020-03-11T22:55:15.599 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:20.277748+0000" 2020-03-11T22:55:15.599 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.599 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.599 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:55:15.599 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:20.007103+0000", 2020-03-11T22:55:15.599 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:20.277759+0000" 2020-03-11T22:55:15.599 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.599 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.600 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:55:15.600 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:20.277962+0000", 2020-03-11T22:55:15.600 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:20.394962+0000" 2020-03-11T22:55:15.600 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.600 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.600 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-03-11T22:55:15.601 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:20.394964+0000", 2020-03-11T22:55:15.601 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:20.395095+0000" 2020-03-11T22:55:15.601 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.601 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.601 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-03-11T22:55:15.601 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:20.395095+0000", 2020-03-11T22:55:15.601 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:20.395443+0000" 2020-03-11T22:55:15.602 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.602 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.602 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Backfilling", 2020-03-11T22:55:15.602 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:20.395443+0000", 2020-03-11T22:55:15.602 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:52.654866+0000" 2020-03-11T22:55:15.602 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.602 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.603 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/NotBackfilling", 2020-03-11T22:55:15.603 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:52.654867+0000", 2020-03-11T22:55:15.603 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:52.655075+0000" 2020-03-11T22:55:15.603 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.603 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.604 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-03-11T22:55:15.604 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:52.655076+0000", 2020-03-11T22:55:15.604 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:54:04.686826+0000" 2020-03-11T22:55:15.604 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.604 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.605 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-03-11T22:55:15.605 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:54:04.686826+0000", 2020-03-11T22:55:15.605 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:54:04.687274+0000" 2020-03-11T22:55:15.605 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.605 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.605 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Backfilling", 2020-03-11T22:55:15.605 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:54:04.687274+0000", 2020-03-11T22:55:15.606 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:54:30.074146+0000" 2020-03-11T22:55:15.606 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.606 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.606 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:55:15.606 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:54:30.074146+0000", 2020-03-11T22:55:15.606 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:54:30.074302+0000" 2020-03-11T22:55:15.607 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.607 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.607 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:55:15.607 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:54:30.074302+0000", 2020-03-11T22:55:15.607 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:54:30.195062+0000" 2020-03-11T22:55:15.607 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.607 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.607 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active", 2020-03-11T22:55:15.608 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:20.277759+0000", 2020-03-11T22:55:15.608 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:54:30.195094+0000" 2020-03-11T22:55:15.608 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.608 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.608 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary", 2020-03-11T22:55:15.608 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:20.007098+0000", 2020-03-11T22:55:15.608 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:54:30.195115+0000" 2020-03-11T22:55:15.609 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.609 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.609 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started", 2020-03-11T22:55:15.609 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:20.007078+0000", 2020-03-11T22:55:15.609 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:54:30.195121+0000" 2020-03-11T22:55:15.609 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:15.609 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:55:15.609 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.610 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.610 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "116", 2020-03-11T22:55:15.610 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2020-03-11T22:55:15.610 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.610 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2020-03-11T22:55:15.610 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:54:30.195122+0000", 2020-03-11T22:55:15.611 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:54:30.195413+0000" 2020-03-11T22:55:15.611 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:15.611 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:55:15.611 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:15.611 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:55:15.611 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.611 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.611 INFO:tasks.workunit.client.0.smithi145.stdout: "pg": "3.0", 2020-03-11T22:55:15.611 INFO:tasks.workunit.client.0.smithi145.stdout: "currently": "Started/Primary/Active/Clean", 2020-03-11T22:55:15.612 INFO:tasks.workunit.client.0.smithi145.stdout: "history": [ 2020-03-11T22:55:15.612 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.612 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "41", 2020-03-11T22:55:15.612 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2020-03-11T22:55:15.613 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.613 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Start", 2020-03-11T22:55:15.613 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:46.979119+0000", 2020-03-11T22:55:15.613 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:46.979131+0000" 2020-03-11T22:55:15.613 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.613 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.614 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:55:15.614 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:46.979139+0000", 2020-03-11T22:55:15.614 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:47.054079+0000" 2020-03-11T22:55:15.614 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.614 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.614 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:55:15.614 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:47.054080+0000", 2020-03-11T22:55:15.614 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:47.054134+0000" 2020-03-11T22:55:15.615 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.615 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.615 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:55:15.615 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:47.054134+0000", 2020-03-11T22:55:15.615 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:47.054145+0000" 2020-03-11T22:55:15.615 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.615 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.616 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:55:15.616 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:47.054146+0000", 2020-03-11T22:55:15.616 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:47.244276+0000" 2020-03-11T22:55:15.616 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.616 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.616 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:55:15.616 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:46.979134+0000", 2020-03-11T22:55:15.616 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:47.244292+0000" 2020-03-11T22:55:15.616 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.617 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.617 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:55:15.617 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:47.244568+0000", 2020-03-11T22:55:15.617 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:47.445873+0000" 2020-03-11T22:55:15.617 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.617 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.618 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:55:15.618 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:47.445873+0000", 2020-03-11T22:55:15.618 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:47.445919+0000" 2020-03-11T22:55:15.618 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.618 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.618 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:55:15.620 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:47.445919+0000", 2020-03-11T22:55:15.620 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:48.580402+0000" 2020-03-11T22:55:15.620 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.620 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.620 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active", 2020-03-11T22:55:15.621 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:47.244292+0000", 2020-03-11T22:55:15.621 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:48.580434+0000" 2020-03-11T22:55:15.621 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.621 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.621 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary", 2020-03-11T22:55:15.622 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:46.979132+0000", 2020-03-11T22:55:15.622 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:48.580461+0000" 2020-03-11T22:55:15.622 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.622 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.622 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started", 2020-03-11T22:55:15.622 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:46.979116+0000", 2020-03-11T22:55:15.623 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:48.580470+0000" 2020-03-11T22:55:15.623 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:15.623 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:55:15.623 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.623 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.623 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "42", 2020-03-11T22:55:15.623 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2020-03-11T22:55:15.624 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.624 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2020-03-11T22:55:15.624 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:48.580471+0000", 2020-03-11T22:55:15.624 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:48.580804+0000" 2020-03-11T22:55:15.624 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:15.624 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:55:15.624 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.624 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.625 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "91", 2020-03-11T22:55:15.625 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2020-03-11T22:55:15.625 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.625 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Start", 2020-03-11T22:55:15.625 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:48.580813+0000", 2020-03-11T22:55:15.625 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:48.580839+0000" 2020-03-11T22:55:15.625 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.626 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.626 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:55:15.626 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:48.580851+0000", 2020-03-11T22:55:15.626 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:48.656439+0000" 2020-03-11T22:55:15.626 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.626 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.626 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:55:15.627 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:48.656440+0000", 2020-03-11T22:55:15.627 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:48.656562+0000" 2020-03-11T22:55:15.627 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.627 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.627 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:55:15.627 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:48.656562+0000", 2020-03-11T22:55:15.627 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:48.656588+0000" 2020-03-11T22:55:15.628 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.628 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.628 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:55:15.628 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:48.656589+0000", 2020-03-11T22:55:15.628 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:48.826274+0000" 2020-03-11T22:55:15.628 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.628 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.628 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:55:15.629 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:48.580847+0000", 2020-03-11T22:55:15.629 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:48.826285+0000" 2020-03-11T22:55:15.629 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.629 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.629 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:55:15.629 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:48.826425+0000", 2020-03-11T22:55:15.630 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:49.057833+0000" 2020-03-11T22:55:15.630 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.630 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.630 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:55:15.630 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:49.057834+0000", 2020-03-11T22:55:15.630 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:49.057919+0000" 2020-03-11T22:55:15.630 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.631 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.631 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:55:15.631 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:49.057919+0000", 2020-03-11T22:55:15.631 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:52:22.713926+0000" 2020-03-11T22:55:15.631 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.631 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.631 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active", 2020-03-11T22:55:15.632 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:48.826286+0000", 2020-03-11T22:55:15.632 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:52:22.713960+0000" 2020-03-11T22:55:15.632 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.632 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.632 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary", 2020-03-11T22:55:15.632 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:48.580840+0000", 2020-03-11T22:55:15.632 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:52:22.713988+0000" 2020-03-11T22:55:15.633 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.633 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.633 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started", 2020-03-11T22:55:15.633 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:48.580805+0000", 2020-03-11T22:55:15.633 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:52:22.713996+0000" 2020-03-11T22:55:15.633 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:15.634 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:55:15.634 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.634 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.634 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "92", 2020-03-11T22:55:15.634 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2020-03-11T22:55:15.634 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.634 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2020-03-11T22:55:15.635 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:52:22.713997+0000", 2020-03-11T22:55:15.635 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:52:22.714306+0000" 2020-03-11T22:55:15.635 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:15.635 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:55:15.635 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.635 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.635 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "108", 2020-03-11T22:55:15.635 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2020-03-11T22:55:15.636 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.636 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Start", 2020-03-11T22:55:15.636 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:52:22.714310+0000", 2020-03-11T22:55:15.636 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:52:22.714324+0000" 2020-03-11T22:55:15.636 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.636 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.636 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:55:15.637 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:52:22.714339+0000", 2020-03-11T22:55:15.637 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:52:22.729818+0000" 2020-03-11T22:55:15.637 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.637 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.637 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:55:15.637 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:52:22.729819+0000", 2020-03-11T22:55:15.637 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:52:22.729907+0000" 2020-03-11T22:55:15.638 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.638 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.638 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:55:15.638 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:52:22.729907+0000", 2020-03-11T22:55:15.638 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:52:22.729929+0000" 2020-03-11T22:55:15.638 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.638 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.639 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:55:15.639 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:52:22.729929+0000", 2020-03-11T22:55:15.639 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:52:22.957830+0000" 2020-03-11T22:55:15.639 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.639 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.639 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:55:15.640 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:52:22.714332+0000", 2020-03-11T22:55:15.640 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:52:22.957838+0000" 2020-03-11T22:55:15.640 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.640 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.640 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:55:15.640 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:52:22.957900+0000", 2020-03-11T22:55:15.640 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:52:22.981679+0000" 2020-03-11T22:55:15.640 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.641 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.641 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:55:15.641 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:52:22.981679+0000", 2020-03-11T22:55:15.641 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:52:22.981717+0000" 2020-03-11T22:55:15.641 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.641 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.641 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:55:15.642 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:52:22.981717+0000", 2020-03-11T22:55:15.642 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:39.175562+0000" 2020-03-11T22:55:15.642 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.642 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.642 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active", 2020-03-11T22:55:15.642 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:52:22.957838+0000", 2020-03-11T22:55:15.642 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:39.175591+0000" 2020-03-11T22:55:15.643 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.643 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.643 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary", 2020-03-11T22:55:15.643 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:52:22.714324+0000", 2020-03-11T22:55:15.643 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:39.175636+0000" 2020-03-11T22:55:15.643 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.643 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.643 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started", 2020-03-11T22:55:15.644 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:52:22.714306+0000", 2020-03-11T22:55:15.644 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:39.175642+0000" 2020-03-11T22:55:15.644 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:15.644 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:55:15.644 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.644 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.644 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "109", 2020-03-11T22:55:15.645 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2020-03-11T22:55:15.645 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.645 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2020-03-11T22:55:15.645 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:39.175642+0000", 2020-03-11T22:55:15.645 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:39.175968+0000" 2020-03-11T22:55:15.645 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:15.645 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:55:15.646 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.646 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.646 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "110", 2020-03-11T22:55:15.646 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2020-03-11T22:55:15.646 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.646 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Start", 2020-03-11T22:55:15.646 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:39.175975+0000", 2020-03-11T22:55:15.647 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:39.175988+0000" 2020-03-11T22:55:15.647 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.647 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.649 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:55:15.650 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:39.175997+0000", 2020-03-11T22:55:15.650 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:39.176396+0000" 2020-03-11T22:55:15.650 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.650 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.650 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:55:15.650 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:39.176396+0000", 2020-03-11T22:55:15.651 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:39.176451+0000" 2020-03-11T22:55:15.651 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.651 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.651 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:55:15.651 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:39.175993+0000", 2020-03-11T22:55:15.651 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:39.176460+0000" 2020-03-11T22:55:15.651 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.652 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.652 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/WaitActingChange", 2020-03-11T22:55:15.652 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:39.176461+0000", 2020-03-11T22:55:15.652 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:39.337501+0000" 2020-03-11T22:55:15.652 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.652 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.652 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary", 2020-03-11T22:55:15.652 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:39.175989+0000", 2020-03-11T22:55:15.653 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:39.337520+0000" 2020-03-11T22:55:15.653 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.653 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.653 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started", 2020-03-11T22:55:15.653 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:39.175969+0000", 2020-03-11T22:55:15.653 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:39.337526+0000" 2020-03-11T22:55:15.653 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:15.654 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:55:15.654 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.654 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.654 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "110", 2020-03-11T22:55:15.654 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2020-03-11T22:55:15.654 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.654 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2020-03-11T22:55:15.655 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:39.337526+0000", 2020-03-11T22:55:15.655 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:39.337727+0000" 2020-03-11T22:55:15.655 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:15.655 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:55:15.655 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.655 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.655 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "118", 2020-03-11T22:55:15.656 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2020-03-11T22:55:15.656 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.656 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Start", 2020-03-11T22:55:15.656 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:39.337731+0000", 2020-03-11T22:55:15.656 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:39.337743+0000" 2020-03-11T22:55:15.656 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.656 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.656 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:55:15.657 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:39.337750+0000", 2020-03-11T22:55:15.657 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:39.409404+0000" 2020-03-11T22:55:15.657 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.657 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.657 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:55:15.657 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:39.409405+0000", 2020-03-11T22:55:15.658 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:39.409515+0000" 2020-03-11T22:55:15.658 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.658 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.658 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:55:15.658 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:39.409515+0000", 2020-03-11T22:55:15.658 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:39.409546+0000" 2020-03-11T22:55:15.658 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.659 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.659 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:55:15.659 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:39.409546+0000", 2020-03-11T22:55:15.659 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:39.568377+0000" 2020-03-11T22:55:15.659 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.659 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.659 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:55:15.659 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:39.337747+0000", 2020-03-11T22:55:15.660 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:39.568388+0000" 2020-03-11T22:55:15.660 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.660 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.660 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:55:15.660 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:39.568763+0000", 2020-03-11T22:55:15.660 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:39.658771+0000" 2020-03-11T22:55:15.660 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.661 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.661 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-03-11T22:55:15.661 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:39.658772+0000", 2020-03-11T22:55:15.661 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:52.654939+0000" 2020-03-11T22:55:15.661 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.661 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.661 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-03-11T22:55:15.661 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:52.654939+0000", 2020-03-11T22:55:15.662 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:52.655321+0000" 2020-03-11T22:55:15.662 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.662 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.662 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Backfilling", 2020-03-11T22:55:15.662 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:52.655322+0000", 2020-03-11T22:55:15.662 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:54:04.686741+0000" 2020-03-11T22:55:15.662 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.663 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.663 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/NotBackfilling", 2020-03-11T22:55:15.664 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:54:04.686742+0000", 2020-03-11T22:55:15.664 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:54:04.686967+0000" 2020-03-11T22:55:15.664 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.664 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.664 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-03-11T22:55:15.664 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:54:04.686967+0000", 2020-03-11T22:55:15.664 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:54:30.074474+0000" 2020-03-11T22:55:15.665 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.665 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.665 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-03-11T22:55:15.665 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:54:30.074474+0000", 2020-03-11T22:55:15.665 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:54:30.074910+0000" 2020-03-11T22:55:15.665 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.665 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.666 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Backfilling", 2020-03-11T22:55:15.666 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:54:30.074911+0000", 2020-03-11T22:55:15.666 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:54:50.404471+0000" 2020-03-11T22:55:15.666 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.666 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.666 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:55:15.666 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:54:50.404472+0000", 2020-03-11T22:55:15.667 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:54:50.404679+0000" 2020-03-11T22:55:15.667 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.667 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.667 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:55:15.667 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:54:50.404680+0000", 2020-03-11T22:55:15.667 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:54:50.541599+0000" 2020-03-11T22:55:15.667 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.668 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.668 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active", 2020-03-11T22:55:15.668 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:39.568388+0000", 2020-03-11T22:55:15.668 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:54:50.541625+0000" 2020-03-11T22:55:15.668 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.668 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.668 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary", 2020-03-11T22:55:15.669 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:39.337743+0000", 2020-03-11T22:55:15.669 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:54:50.541644+0000" 2020-03-11T22:55:15.669 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.669 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.669 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started", 2020-03-11T22:55:15.669 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:39.337728+0000", 2020-03-11T22:55:15.669 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:54:50.541650+0000" 2020-03-11T22:55:15.670 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:15.670 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:55:15.670 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.670 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.670 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "118", 2020-03-11T22:55:15.670 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2020-03-11T22:55:15.670 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.671 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2020-03-11T22:55:15.671 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:54:50.541651+0000", 2020-03-11T22:55:15.671 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:54:50.541929+0000" 2020-03-11T22:55:15.671 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:15.671 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:55:15.671 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:15.671 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:55:15.671 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.672 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.672 INFO:tasks.workunit.client.0.smithi145.stdout: "pg": "1.0", 2020-03-11T22:55:15.672 INFO:tasks.workunit.client.0.smithi145.stdout: "currently": "Started/Primary/Active/Clean", 2020-03-11T22:55:15.672 INFO:tasks.workunit.client.0.smithi145.stdout: "history": [ 2020-03-11T22:55:15.672 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.672 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "27", 2020-03-11T22:55:15.672 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2020-03-11T22:55:15.673 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.673 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Initial", 2020-03-11T22:55:15.673 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:43.140910+0000", 2020-03-11T22:55:15.673 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:43.141063+0000" 2020-03-11T22:55:15.673 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:15.673 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:55:15.673 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.674 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.674 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "27", 2020-03-11T22:55:15.674 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2020-03-11T22:55:15.674 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.674 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2020-03-11T22:55:15.674 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:43.141063+0000", 2020-03-11T22:55:15.674 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:43.141101+0000" 2020-03-11T22:55:15.674 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:15.675 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:55:15.675 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.675 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.675 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "29", 2020-03-11T22:55:15.675 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2020-03-11T22:55:15.675 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.675 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Start", 2020-03-11T22:55:15.676 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:43.141105+0000", 2020-03-11T22:55:15.676 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:43.141117+0000" 2020-03-11T22:55:15.676 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.676 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.676 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:55:15.676 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:43.141125+0000", 2020-03-11T22:55:15.676 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:43.158099+0000" 2020-03-11T22:55:15.677 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.677 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.677 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:55:15.677 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:43.158099+0000", 2020-03-11T22:55:15.677 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:43.158191+0000" 2020-03-11T22:55:15.677 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.677 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.678 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:55:15.678 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:43.158191+0000", 2020-03-11T22:55:15.678 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:43.158213+0000" 2020-03-11T22:55:15.678 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.678 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.678 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:55:15.678 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:43.158214+0000", 2020-03-11T22:55:15.679 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:43.303318+0000" 2020-03-11T22:55:15.679 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.679 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.679 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:55:15.679 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:43.141121+0000", 2020-03-11T22:55:15.679 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:43.303334+0000" 2020-03-11T22:55:15.679 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.679 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.680 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:55:15.680 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:43.303643+0000", 2020-03-11T22:55:15.680 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:43.845042+0000" 2020-03-11T22:55:15.680 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.680 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.680 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active", 2020-03-11T22:55:15.680 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:43.303335+0000", 2020-03-11T22:55:15.681 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:43.845067+0000" 2020-03-11T22:55:15.681 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.681 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.681 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary", 2020-03-11T22:55:15.681 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:43.141117+0000", 2020-03-11T22:55:15.681 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:43.845081+0000" 2020-03-11T22:55:15.681 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.682 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.682 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started", 2020-03-11T22:55:15.682 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:43.141102+0000", 2020-03-11T22:55:15.682 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:43.845088+0000" 2020-03-11T22:55:15.682 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:15.682 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:55:15.682 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.683 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.683 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "29", 2020-03-11T22:55:15.683 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2020-03-11T22:55:15.683 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.683 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2020-03-11T22:55:15.683 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:43.845088+0000", 2020-03-11T22:55:15.683 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:43.845290+0000" 2020-03-11T22:55:15.684 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:15.684 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:55:15.684 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.684 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.684 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "104", 2020-03-11T22:55:15.684 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2020-03-11T22:55:15.684 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.685 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Start", 2020-03-11T22:55:15.685 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:43.845293+0000", 2020-03-11T22:55:15.685 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:43.845303+0000" 2020-03-11T22:55:15.685 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.685 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.685 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:55:15.685 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:43.845310+0000", 2020-03-11T22:55:15.686 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:43.977269+0000" 2020-03-11T22:55:15.686 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.686 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.686 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:55:15.686 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:43.977269+0000", 2020-03-11T22:55:15.686 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:43.977339+0000" 2020-03-11T22:55:15.686 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.687 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.687 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:55:15.687 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:43.977339+0000", 2020-03-11T22:55:15.687 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:43.977353+0000" 2020-03-11T22:55:15.687 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.687 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.687 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:55:15.687 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:43.977354+0000", 2020-03-11T22:55:15.688 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:44.043682+0000" 2020-03-11T22:55:15.688 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.688 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.688 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:55:15.688 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:43.845306+0000", 2020-03-11T22:55:15.688 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:44.043692+0000" 2020-03-11T22:55:15.688 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.689 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.689 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:55:15.689 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:44.043829+0000", 2020-03-11T22:55:15.689 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:44.073728+0000" 2020-03-11T22:55:15.689 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.689 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.689 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:55:15.690 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:44.073729+0000", 2020-03-11T22:55:15.690 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:51:44.073775+0000" 2020-03-11T22:55:15.690 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.690 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.690 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:55:15.690 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:44.073775+0000", 2020-03-11T22:55:15.690 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:29.955148+0000" 2020-03-11T22:55:15.691 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.691 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.691 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active", 2020-03-11T22:55:15.691 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:44.043692+0000", 2020-03-11T22:55:15.691 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:29.955174+0000" 2020-03-11T22:55:15.691 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.691 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.691 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary", 2020-03-11T22:55:15.692 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:43.845303+0000", 2020-03-11T22:55:15.692 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:29.955213+0000" 2020-03-11T22:55:15.692 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.692 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.692 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started", 2020-03-11T22:55:15.692 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:51:43.845290+0000", 2020-03-11T22:55:15.692 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:29.955218+0000" 2020-03-11T22:55:15.693 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:15.693 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:55:15.693 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.693 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.693 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "104", 2020-03-11T22:55:15.693 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2020-03-11T22:55:15.693 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.694 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2020-03-11T22:55:15.694 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:29.955218+0000", 2020-03-11T22:55:15.694 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:29.955500+0000" 2020-03-11T22:55:15.694 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:15.694 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:55:15.694 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.694 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.694 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "106", 2020-03-11T22:55:15.695 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2020-03-11T22:55:15.695 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.695 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Start", 2020-03-11T22:55:15.695 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:29.955506+0000", 2020-03-11T22:55:15.695 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:29.955519+0000" 2020-03-11T22:55:15.695 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.695 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.696 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:55:15.696 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:29.955527+0000", 2020-03-11T22:55:15.696 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:30.017113+0000" 2020-03-11T22:55:15.696 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.696 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.696 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:55:15.696 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:30.017113+0000", 2020-03-11T22:55:15.697 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:30.017181+0000" 2020-03-11T22:55:15.697 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.697 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.697 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:55:15.697 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:29.955523+0000", 2020-03-11T22:55:15.697 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:30.017190+0000" 2020-03-11T22:55:15.697 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.698 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.698 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/WaitActingChange", 2020-03-11T22:55:15.698 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:30.017190+0000", 2020-03-11T22:55:15.698 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:30.305408+0000" 2020-03-11T22:55:15.698 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.698 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.698 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary", 2020-03-11T22:55:15.698 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:29.955520+0000", 2020-03-11T22:55:15.699 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:30.305424+0000" 2020-03-11T22:55:15.699 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.699 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.699 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started", 2020-03-11T22:55:15.699 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:29.955500+0000", 2020-03-11T22:55:15.699 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:30.305430+0000" 2020-03-11T22:55:15.699 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:15.700 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:55:15.700 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.700 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.700 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "106", 2020-03-11T22:55:15.700 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2020-03-11T22:55:15.700 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.701 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2020-03-11T22:55:15.701 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:30.305430+0000", 2020-03-11T22:55:15.701 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:30.305644+0000" 2020-03-11T22:55:15.701 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:15.701 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:55:15.701 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.701 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.702 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "120", 2020-03-11T22:55:15.702 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2020-03-11T22:55:15.702 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.702 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Start", 2020-03-11T22:55:15.702 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:30.305648+0000", 2020-03-11T22:55:15.702 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:30.305669+0000" 2020-03-11T22:55:15.702 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.702 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.703 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:55:15.703 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:30.305680+0000", 2020-03-11T22:55:15.703 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:30.330731+0000" 2020-03-11T22:55:15.703 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.703 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.703 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:55:15.703 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:30.330732+0000", 2020-03-11T22:55:15.704 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:30.330820+0000" 2020-03-11T22:55:15.704 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.704 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.704 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:55:15.704 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:30.330821+0000", 2020-03-11T22:55:15.704 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:30.330844+0000" 2020-03-11T22:55:15.704 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.704 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.705 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:55:15.705 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:30.330845+0000", 2020-03-11T22:55:15.705 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:30.513618+0000" 2020-03-11T22:55:15.705 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.705 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.705 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:55:15.705 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:30.305675+0000", 2020-03-11T22:55:15.706 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:30.513648+0000" 2020-03-11T22:55:15.706 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.706 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.706 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:55:15.706 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:30.514005+0000", 2020-03-11T22:55:15.706 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:53:30.703614+0000" 2020-03-11T22:55:15.706 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.706 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.707 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-03-11T22:55:15.707 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:30.703615+0000", 2020-03-11T22:55:15.707 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:54:50.404905+0000" 2020-03-11T22:55:15.707 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.707 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.707 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-03-11T22:55:15.707 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:54:50.404905+0000", 2020-03-11T22:55:15.708 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:54:50.405352+0000" 2020-03-11T22:55:15.708 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.708 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.708 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Backfilling", 2020-03-11T22:55:15.708 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:54:50.405353+0000", 2020-03-11T22:55:15.708 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:55:10.682230+0000" 2020-03-11T22:55:15.708 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.709 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.709 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:55:15.709 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:55:10.682230+0000", 2020-03-11T22:55:15.709 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:55:10.682412+0000" 2020-03-11T22:55:15.709 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.709 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.709 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:55:15.710 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:55:10.682413+0000", 2020-03-11T22:55:15.710 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:55:10.891025+0000" 2020-03-11T22:55:15.710 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.710 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.710 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active", 2020-03-11T22:55:15.710 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:30.513648+0000", 2020-03-11T22:55:15.710 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:55:10.891050+0000" 2020-03-11T22:55:15.710 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.711 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.711 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary", 2020-03-11T22:55:15.711 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:30.305670+0000", 2020-03-11T22:55:15.711 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:55:10.891069+0000" 2020-03-11T22:55:15.711 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.711 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.711 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started", 2020-03-11T22:55:15.712 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:53:30.305644+0000", 2020-03-11T22:55:15.712 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:55:10.891075+0000" 2020-03-11T22:55:15.712 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:15.712 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:55:15.712 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2020-03-11T22:55:15.712 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.712 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "120", 2020-03-11T22:55:15.712 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2020-03-11T22:55:15.713 INFO:tasks.workunit.client.0.smithi145.stdout: { 2020-03-11T22:55:15.713 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2020-03-11T22:55:15.713 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2020-03-11T22:55:10.891075+0000", 2020-03-11T22:55:15.713 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2020-03-11T22:55:10.891352+0000" 2020-03-11T22:55:15.713 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:15.713 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:55:15.713 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:15.714 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:55:15.714 INFO:tasks.workunit.client.0.smithi145.stdout: } 2020-03-11T22:55:15.714 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2020-03-11T22:55:15.714 INFO:tasks.workunit.client.0.smithi145.stdout:} 2020-03-11T22:55:15.715 INFO:tasks.workunit.client.0.smithi145.stdout:TEST PASSED 2020-03-11T22:55:15.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:302: TEST_backfill_priority: '[' 0 '!=' 0 ']' 2020-03-11T22:55:15.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:306: TEST_backfill_priority: echo TEST PASSED 2020-03-11T22:55:15.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:309: TEST_backfill_priority: delete_pool test1 2020-03-11T22:55:15.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test1 2020-03-11T22:55:15.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2020-03-11T22:55:16.023 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test1' does not exist 2020-03-11T22:55:16.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:310: TEST_backfill_priority: delete_pool test3 2020-03-11T22:55:16.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test3 2020-03-11T22:55:16.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2020-03-11T22:55:16.543 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test3' does not exist 2020-03-11T22:55:16.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:311: TEST_backfill_priority: delete_pool test7 2020-03-11T22:55:16.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test7 2020-03-11T22:55:16.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test7 test7 --yes-i-really-really-mean-it 2020-03-11T22:55:17.086 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test7' does not exist 2020-03-11T22:55:17.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:312: TEST_backfill_priority: kill_daemons td/osd-backfill-prio 2020-03-11T22:55:17.095 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:55:17.095 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:55:17.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:55:17.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:55:17.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:55:17.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:55:17.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:313: TEST_backfill_priority: return 0 2020-03-11T22:55:17.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:40: run: teardown td/osd-backfill-prio 2020-03-11T22:55:17.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-03-11T22:55:17.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:55:17.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-03-11T22:55:17.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:55:17.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:55:17.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:55:17.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:55:17.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:55:17.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:55:17.210 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:55:17.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:55:17.211 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:55:17.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:55:17.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:55:17.212 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:55:17.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:55:17.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:55:17.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:55:17.214 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:55:17.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:55:17.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:55:17.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-03-11T22:55:17.242 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:55:17.242 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:17.242 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56096 2020-03-11T22:55:17.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.56096 2020-03-11T22:55:17.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:55:17.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:55:17.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: code=0 2020-03-11T22:55:17.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: teardown td/osd-backfill-prio 0 2020-03-11T22:55:17.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-03-11T22:55:17.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-03-11T22:55:17.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-03-11T22:55:17.245 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:55:17.245 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:55:17.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:55:17.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:55:17.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:55:17.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:55:17.246 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:55:17.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:55:17.246 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:55:17.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:55:17.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:55:17.247 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:55:17.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:55:17.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:55:17.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:55:17.250 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:55:17.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:55:17.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-03-11T22:55:17.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-03-11T22:55:17.252 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:55:17.252 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:17.252 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56096 2020-03-11T22:55:17.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.56096 2020-03-11T22:55:17.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:55:17.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:55:17.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: return 0 2020-03-11T22:55:17.254 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-03-11T22:55:17.254 INFO:teuthology.orchestra.run.smithi145:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-03-11T22:55:17.333 INFO:tasks.workunit:Running workunit osd/osd-backfill-recovery-log.sh... 2020-03-11T22:55:17.333 INFO:teuthology.orchestra.run.smithi145:workunit test osd/osd-backfill-recovery-log.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4acc1121e47039a3cdf2978ae458c10d153aa141 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh 2020-03-11T22:55:17.391 INFO:tasks.workunit.client.0.smithi145.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-03-11T22:55:17.399 INFO:tasks.workunit.client.0.smithi145.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-03-11T22:55:17.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:55:17.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:55:17.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: export PYTHONWARNINGS=ignore 2020-03-11T22:55:17.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: PYTHONWARNINGS=ignore 2020-03-11T22:55:17.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: export CEPH_CONF=/dev/null 2020-03-11T22:55:17.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: CEPH_CONF=/dev/null 2020-03-11T22:55:17.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: main: unset CEPH_ARGS 2020-03-11T22:55:17.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: local code 2020-03-11T22:55:17.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: run td/osd-backfill-recovery-log 2020-03-11T22:55:17.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:21: run: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:17.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:22: run: shift 2020-03-11T22:55:17.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:25: run: export CEPH_MON=127.0.0.1:7129 2020-03-11T22:55:17.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:25: run: CEPH_MON=127.0.0.1:7129 2020-03-11T22:55:17.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:26: run: export CEPH_ARGS 2020-03-11T22:55:17.401 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:27: run: uuidgen 2020-03-11T22:55:17.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:27: run: CEPH_ARGS+='--fsid=d0b6139e-bf15-4877-a0b2-251bf4ba269f --auth-supported=none ' 2020-03-11T22:55:17.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:55:17.402 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:30: run: set 2020-03-11T22:55:17.402 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-03-11T22:55:17.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:30: run: local 'funcs=TEST_backfill_log_1 2020-03-11T22:55:17.404 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_backfill_log_2 2020-03-11T22:55:17.404 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_recovery_1 2020-03-11T22:55:17.405 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_recovery_2' 2020-03-11T22:55:17.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:31: run: for func in $funcs 2020-03-11T22:55:17.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:32: run: setup td/osd-backfill-recovery-log 2020-03-11T22:55:17.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:17.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-recovery-log 2020-03-11T22:55:17.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:17.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:55:17.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2020-03-11T22:55:17.406 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:55:17.406 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:55:17.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:55:17.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:55:17.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:55:17.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:55:17.406 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:55:17.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:55:17.407 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:55:17.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:55:17.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:55:17.407 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:55:17.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:55:17.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:55:17.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:55:17.409 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:55:17.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:55:17.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:55:17.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2020-03-11T22:55:17.410 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:55:17.410 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:17.410 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76655 2020-03-11T22:55:17.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76655 2020-03-11T22:55:17.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:55:17.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:55:17.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-recovery-log 2020-03-11T22:55:17.411 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:55:17.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:17.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76655 2020-03-11T22:55:17.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76655 2020-03-11T22:55:17.414 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:55:17.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-03-11T22:55:17.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-03-11T22:55:17.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:55:17.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-recovery-log 1' TERM HUP INT 2020-03-11T22:55:17.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:33: run: TEST_backfill_log_1 td/osd-backfill-recovery-log 2020-03-11T22:55:17.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:103: TEST_backfill_log_1: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:17.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:105: TEST_backfill_log_1: _common_test td/osd-backfill-recovery-log '--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2 8 150 2020-03-11T22:55:17.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:40: _common_test: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:17.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:41: _common_test: local 'extra_opts=--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:55:17.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:42: _common_test: local loglen=2 2020-03-11T22:55:17.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:43: _common_test: local dupslen=8 2020-03-11T22:55:17.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:44: _common_test: local objects=150 2020-03-11T22:55:17.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:45: _common_test: local moreobjects=0 2020-03-11T22:55:17.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:47: _common_test: local OSDS=6 2020-03-11T22:55:17.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:49: _common_test: run_mon td/osd-backfill-recovery-log a 2020-03-11T22:55:17.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:17.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:55:17.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:55:17.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:55:17.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-recovery-log/a 2020-03-11T22:55:17.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-recovery-log/a --run-dir=td/osd-backfill-recovery-log 2020-03-11T22:55:17.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:55:17.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:55:17.717 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:55:17.717 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:55:17.717 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:17.717 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76655 2020-03-11T22:55:17.717 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76655/$cluster-$name.asok' 2020-03-11T22:55:17.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-recovery-log/a '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.76655/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-recovery-log/log --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:55:17.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:55:17.868 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:55:17.868 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:55:17.868 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:55:17.869 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:55:17.869 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:55:17.869 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:55:17.869 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:55:17.869 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:55:17.869 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:55:17.870 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:17.870 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76655 2020-03-11T22:55:17.870 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76655/ceph-mon.a.asok 2020-03-11T22:55:17.870 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:55:17.870 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.76655/ceph-mon.a.asok config get fsid 2020-03-11T22:55:17.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:55:17.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:55:17.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:55:17.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:55:17.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:55:17.957 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:55:17.957 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:55:17.957 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:55:17.957 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:55:17.957 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:17.957 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76655 2020-03-11T22:55:17.957 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76655/ceph-mon.a.asok 2020-03-11T22:55:17.958 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:55:17.958 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.76655/ceph-mon.a.asok config get mon_host 2020-03-11T22:55:18.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:50: _common_test: run_mgr td/osd-backfill-recovery-log x 2020-03-11T22:55:18.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:18.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:55:18.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:55:18.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:55:18.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-backfill-recovery-log/x 2020-03-11T22:55:18.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:55:18.917 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:55:18.917 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:55:18.917 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:55:18.917 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:55:18.917 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:18.917 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76655 2020-03-11T22:55:18.918 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76655/$cluster-$name.asok' 2020-03-11T22:55:18.918 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:55:18.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-recovery-log/x '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.76655/$cluster-$name.asok' --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:55:18.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:51: _common_test: export CEPH_ARGS 2020-03-11T22:55:18.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: export 'EXTRA_OPTS= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:55:18.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: EXTRA_OPTS=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:55:18.939 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: expr 6 - 1 2020-03-11T22:55:18.940 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: seq 0 5 2020-03-11T22:55:18.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:55:18.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 0 2020-03-11T22:55:18.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:18.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:55:18.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:55:18.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:55:18.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/0 2020-03-11T22:55:18.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=d0b6139e-bf15-4877-a0b2-251bf4ba269f --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:55:18.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:55:18.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:55:18.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:55:18.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/0' 2020-03-11T22:55:18.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/0/journal' 2020-03-11T22:55:18.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:55:18.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:55:18.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:55:18.951 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:55:18.951 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:55:18.951 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:55:18.951 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:55:18.951 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:18.951 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76655 2020-03-11T22:55:18.951 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76655/$cluster-$name.asok' 2020-03-11T22:55:18.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76655/$cluster-$name.asok' 2020-03-11T22:55:18.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:55:18.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:55:18.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:55:18.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:55:18.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:55:18.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:55:18.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:55:18.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:55:18.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:55:18.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:55:18.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/0 2020-03-11T22:55:18.953 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:55:18.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=869eab82-62ea-4d48-8cf9-e502008d2158 2020-03-11T22:55:18.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 869eab82-62ea-4d48-8cf9-e502008d2158' 2020-03-11T22:55:18.954 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:55:18.954 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 869eab82-62ea-4d48-8cf9-e502008d2158 2020-03-11T22:55:18.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBWbGle6VvtOBAAYQi89BTekvIusPgBmPrWHA== 2020-03-11T22:55:18.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBWbGle6VvtOBAAYQi89BTekvIusPgBmPrWHA=="}' 2020-03-11T22:55:18.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 869eab82-62ea-4d48-8cf9-e502008d2158 -i td/osd-backfill-recovery-log/0/new.json 2020-03-11T22:55:19.290 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:55:19.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/0/new.json 2020-03-11T22:55:19.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=d0b6139e-bf15-4877-a0b2-251bf4ba269f --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76655/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBWbGle6VvtOBAAYQi89BTekvIusPgBmPrWHA== --osd-uuid 869eab82-62ea-4d48-8cf9-e502008d2158 2020-03-11T22:55:19.324 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:55:19.321+0000 7f106a756ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:19.326 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:55:19.323+0000 7f106a756ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:19.327 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:55:19.324+0000 7f106a756ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:19.328 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:55:19.325+0000 7f106a756ec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:55:19.328 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:55:19.325+0000 7f106a756ec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:55:19.328 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:55:19.325+0000 7f106a756ec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:55:19.366 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:55:19.363+0000 7f106a756ec0 -1 bluestore(td/osd-backfill-recovery-log/0) _read_fsid unparsable uuid 2020-03-11T22:55:22.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/0/keyring 2020-03-11T22:55:22.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:55:22.324 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2020-03-11T22:55:22.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:55:22.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:55:22.777 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2020-03-11T22:55:22.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:55:22.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=d0b6139e-bf15-4877-a0b2-251bf4ba269f --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76655/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:55:22.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:55:22.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:55:22.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:55:22.799 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:55:22.796+0000 7f1726adcec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:22.800 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:55:22.797+0000 7f1726adcec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:22.803 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:55:22.800+0000 7f1726adcec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:23.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:55:23.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:55:23.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:55:23.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:55:23.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:55:23.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:55:23.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:55:23.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:55:23.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:55:23.123 INFO:tasks.workunit.client.0.smithi145.stdout:0 2020-03-11T22:55:23.317 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:55:23.314+0000 7f1726adcec0 -1 Falling back to public interface 2020-03-11T22:55:23.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:55:24.073 INFO:tasks.workunit.client.0.smithi145.stderr:2020-03-11T22:55:24.070+0000 7f1726adcec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:55:24.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:55:24.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:55:24.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:55:24.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:55:24.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:55:24.469 INFO:tasks.workunit.client.0.smithi145.stdout:1 2020-03-11T22:55:24.825 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/77087,v1:127.0.0.1:6803/77087] [v2:127.0.0.1:6804/77087,v1:127.0.0.1:6805/77087] exists,up 869eab82-62ea-4d48-8cf9-e502008d2158 2020-03-11T22:55:24.825 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 f3dff1d9-0226-4ed8-b270-969ef5ca9840 2020-03-11T22:55:24.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:55:24.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:55:24.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:55:24.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:55:24.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 1 2020-03-11T22:55:24.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:24.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:55:24.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:55:24.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:55:24.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/1 2020-03-11T22:55:24.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=d0b6139e-bf15-4877-a0b2-251bf4ba269f --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:55:24.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:55:24.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:55:24.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:55:24.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/1' 2020-03-11T22:55:24.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/1/journal' 2020-03-11T22:55:24.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:55:24.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:55:24.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:55:24.828 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:55:24.828 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:55:24.829 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:55:24.829 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:55:24.829 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:24.829 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76655 2020-03-11T22:55:24.829 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76655/$cluster-$name.asok' 2020-03-11T22:55:24.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76655/$cluster-$name.asok' 2020-03-11T22:55:24.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:55:24.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:55:24.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:55:24.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:55:24.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:55:24.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:55:24.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:55:24.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:55:24.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:55:24.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:55:24.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/1 2020-03-11T22:55:24.831 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:55:24.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=f3dff1d9-0226-4ed8-b270-969ef5ca9840 2020-03-11T22:55:24.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 f3dff1d9-0226-4ed8-b270-969ef5ca9840' 2020-03-11T22:55:24.832 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:55:24.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBcbGleyhfJMRAAdvJRIo7AKufAGZVu+UX6pg== 2020-03-11T22:55:24.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBcbGleyhfJMRAAdvJRIo7AKufAGZVu+UX6pg=="}' 2020-03-11T22:55:24.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new f3dff1d9-0226-4ed8-b270-969ef5ca9840 -i td/osd-backfill-recovery-log/1/new.json 2020-03-11T22:55:25.013 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2020-03-11T22:55:25.013 DEBUG:teuthology.task.console_log:Killing console logger for smithi145 2020-03-11T22:55:25.014 DEBUG:teuthology.task.console_log:Killing console logger for smithi145 2020-03-11T22:55:25.016 DEBUG:teuthology.exit:Finished running handlers